Skip to content

feat(processor): merge ug- AP/NN into consolidated ContainerProfile#319

Open
yugal07 wants to merge 2 commits into
kubescape:mainfrom
yugal07:feat/315-user-managed-profile-merge
Open

feat(processor): merge ug- AP/NN into consolidated ContainerProfile#319
yugal07 wants to merge 2 commits into
kubescape:mainfrom
yugal07:feat/315-user-managed-profile-merge

Conversation

@yugal07
Copy link
Copy Markdown

@yugal07 yugal07 commented May 7, 2026

Server-side aggregation of user-managed (ug-prefix) ApplicationProfile and
NetworkNeighborhood into the consolidated ContainerProfile during
consolidateKeyTimeSeries. Lets node-agent treat the CP as a single source
of truth and drop its per-tick ug- fetch.

  • Merge math ported from node-agent's projection.go, operating on the
    internal softwarecomposition types (no interface or storage method
  • Idempotent across ticks via kubescape.io/last-merged-ug-{ap,nn}-rv
    annotation markers; merge skipped when ug- ResourceVersion is unchanged.
  • Re-consolidation trigger: option (c) (Aggregate user-managed (ug- prefix) profiles into the consolidated ContainerProfile #315 discussion), relies on the existing 30s
    ConsolidateTimeSeries cadence; no AP/NN watcher added (per @matthyx
    guidance, memory cost of caching ug- objects rules out a watch path).
  • Deterministic ordering for DNSNames and MatchExpressions to keep
    SyncChecksum stable across re-merges of unchanged content.
  • LabelSelector / PodSelector / NamespaceSelector MatchLabels: user value
    wins on collision (utils.MergeMaps preserves base, so a local
    overrideMerge is used here instead).

Refs #315, kubescape/node-agent#788

Known limitations (to discuss in PR review):

  • Stale-on-edit/delete: merge is additive against the persisted CP, so
    removing entries from a ug- CRD does not retract them from already-
    merged consolidated CPs. The proper fix is a CRD shape change
    (separate user-managed sub-spec, replaced wholesale)
  • Idle workloads: updateProfile early-returns when no new time-series
    arrives, so ug- edits to Completed workloads do not propagate until
    the next tick that has new data. Accepted trade-off for option (c).

worth a note: Changes were initially drafted using help from copilot, and manually reviewed and edited

Summary by CodeRabbit

Release Notes

  • New Features

    • User-managed application and network profiles are now consolidated into container profiles, with idempotent merging that prevents duplicate entries when consolidation is repeated without source data changes.
  • Tests

    • Added comprehensive test coverage for user-managed profile consolidation, including merge validation, idempotency verification, and container-specific data routing.

Signed-off-by: yugal07 <yashsadhwani544@gmail.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 7, 2026

Review Change Stack

Warning

Rate limit exceeded

@yugal07 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 15 minutes and 11 seconds before requesting another review.

To continue reviewing without waiting, purchase usage credits in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1f2b59d4-18b8-429f-b665-cfafc6649deb

📥 Commits

Reviewing files that changed from the base of the PR and between 884d791 and 606934c.

📒 Files selected for processing (2)
  • pkg/registry/file/containerprofile_user_managed.go
  • pkg/registry/file/containerprofile_user_managed_test.go
📝 Walkthrough

Walkthrough

This PR adds server-side consolidation of user-managed ApplicationProfile and NetworkNeighborhood objects into ContainerProfiles during processing. The processor now fetches matching user-managed objects from SQLite storage, merges per-container entries using annotation-based idempotency markers, and persists the consolidated result.

Changes

User-Managed Profile Consolidation

Layer / File(s) Summary
User-Managed Merge Orchestration
pkg/registry/file/containerprofile_user_managed.go
mergeUserManagedProfiles derives workload/container identity from InstanceID annotations, fetches ApplicationProfile and NetworkNeighborhood from SQLite with 5s timeouts, and conditionally merges per-container entries into profile.Spec using ResourceVersion markers for idempotency.
Storage Access & Guards
pkg/registry/file/containerprofile_user_managed.go
userManagedConn accessor extracts StorageImpl and SQLite connection from context, warns once and disables merge when backend is not SQLite or connection is unavailable.
Container Profile Merge Implementations
pkg/registry/file/containerprofile_user_managed.go
mergeUserAPIntoCP and mergeUserNNIntoCP find matching containers across normal/init/ephemeral slices, deep-copy user data, append/merge capabilities/syscalls/endpoints with mergePolicies collision resolution, and merge NetworkNeighborhood ingress/egress by identifier with deterministic label selector deduplication.
Processor UpdateProfile Integration
pkg/registry/file/containerprofile_processor.go
updateProfile exits early only when there is no new data or missing InstanceID annotation; otherwise calls mergeUserManagedProfiles before initialization timestamp and persistence.
Unit Test Coverage
pkg/registry/file/containerprofile_user_managed_test.go
Unit tests validate ApplicationProfile merging (capabilities/syscalls union, PolicyByRuleId entry merge, init/ephemeral container matching, no-alias semantics) and NetworkNeighborhood merging (ingress union by identifier, DNS list merging, port collision semantics, label selector override behavior).
E2E Test Harness & Validation
pkg/registry/file/containerprofile_user_managed_test.go
E2E tests using SQLite storage and ContainerProfileProcessor validate user-managed entry merging, ResourceVersion marker advancement on object updates, idempotency across repeated consolidation ticks, status/completion annotation preservation from time-series flow, and correct fan-out of container-specific entries to per-container CPs without cross-contamination.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

Poem

🐰 A merging deed in SQLite's keep,
User-managed profiles blend deep,
Idempotent marks ensure no strife,
Container tales are unified life! 🎯

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 46.88% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: merging user-managed (ug-) ApplicationProfile and NetworkNeighborhood resources into consolidated ContainerProfile during processing.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@yugal07
Copy link
Copy Markdown
Author

yugal07 commented May 7, 2026

@matthyx. Let me know what all things needs to change. 😄

@matthyx
Copy link
Copy Markdown
Contributor

matthyx commented May 7, 2026

@matthyx. Let me know what all things needs to change. 😄

I'll try to review it tomorrow, it's a lot of mental load

@yugal07
Copy link
Copy Markdown
Author

yugal07 commented May 7, 2026

@matthyx. Let me know what all things needs to change. 😄

I'll try to review it tomorrow, it's a lot of mental load

No issues. Thanks

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
pkg/registry/file/containerprofile_user_managed.go (2)

211-212: 💤 Low value

append(base, user...) aliases base's backing array when cap(base) > len(base).

The subsequent out := combined[:0] in-place filter is algorithmically correct (since len(out) <= i holds at every step, so combined[i] is always read before being overwritten). However, the shared backing array with base makes the invariant non-obvious and fragile for future maintainers. An explicit allocation costs O(n) but makes the intent clear:

♻️ Proposed refactor
-combined := append(base, user...)
+combined := make([]metav1.LabelSelectorRequirement, 0, len(base)+len(user))
+combined = append(combined, base...)
+combined = append(combined, user...)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/registry/file/containerprofile_user_managed.go` around lines 211 - 212,
appendDedupSortedMatchExpressions currently does combined := append(base,
user...) which can alias base's backing array; change to allocate a new slice of
size len(base)+len(user), copy base then user into it so combined has its own
backing array before the in-place filter (out := combined[:0]) — this ensures
the subsequent overwrite loop cannot accidentally mutate the original base
slice.

128-131: 💤 Low value

Missing observability when conn is absent from context.

The type-assertion miss on lines 121–126 logs a Warning via userManagedConnWarnOnce, but the conn-missing path (lines 128–131) returns ok=false silently. If WithConnection is somehow not called upstream or the context key changes, the ug- merge is silently disabled with no production signal.

🔍 Proposed addition
 conn, ok := ctx.Value(connKey).(*sqlite.Conn)
 if !ok {
+    userManagedConnWarnOnce.Do(func() {
+        logger.L().Warning("ContainerProfileProcessor.mergeUserManagedProfiles disabled - sqlite connection not found in context")
+    })
     return nil, nil, false
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/registry/file/containerprofile_user_managed.go` around lines 128 - 131,
The type-assertion failure branch that currently returns (nil, nil, false) when
conn := ctx.Value(connKey).(*sqlite.Conn) fails should emit the same observable
warning as the earlier path; update the missing-conn path in
containerprofile_user_managed.go to call the existing userManagedConnWarnOnce
warning (or the same logger used there) before returning so absence of
WithConnection/connKey is logged in production; reference connKey, ctx.Value,
sqlite.Conn and userManagedConnWarnOnce to locate and patch the branch.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/registry/file/containerprofile_user_managed_test.go`:
- Around line 262-289: The ContainerProfileProcessor in the test harness is
created with an empty HostType which relies on implicit equivalence to
HostTypeKubernetes; update newE2EHarness (and similarly
TestConsolidateUserManagedFanOut) to set HostType: armotypes.HostTypeKubernetes
when constructing the ContainerProfileProcessor so the test harness mirrors
production/NewContainerProfileProcessor behavior and avoids relying on
ParseContainerProfileKey/BuildContainerProfileKey implicit handling of empty
HostType.

---

Nitpick comments:
In `@pkg/registry/file/containerprofile_user_managed.go`:
- Around line 211-212: appendDedupSortedMatchExpressions currently does combined
:= append(base, user...) which can alias base's backing array; change to
allocate a new slice of size len(base)+len(user), copy base then user into it so
combined has its own backing array before the in-place filter (out :=
combined[:0]) — this ensures the subsequent overwrite loop cannot accidentally
mutate the original base slice.
- Around line 128-131: The type-assertion failure branch that currently returns
(nil, nil, false) when conn := ctx.Value(connKey).(*sqlite.Conn) fails should
emit the same observable warning as the earlier path; update the missing-conn
path in containerprofile_user_managed.go to call the existing
userManagedConnWarnOnce warning (or the same logger used there) before returning
so absence of WithConnection/connKey is logged in production; reference connKey,
ctx.Value, sqlite.Conn and userManagedConnWarnOnce to locate and patch the
branch.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ea6679fe-673c-4568-879e-43738d630294

📥 Commits

Reviewing files that changed from the base of the PR and between 2a689d8 and 884d791.

📒 Files selected for processing (3)
  • pkg/registry/file/containerprofile_processor.go
  • pkg/registry/file/containerprofile_user_managed.go
  • pkg/registry/file/containerprofile_user_managed_test.go

Comment thread pkg/registry/file/containerprofile_user_managed_test.go
Signed-off-by: yugal07 <yashsadhwani544@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants