Skip to content

Continuous Profiling Implementation #282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 10 commits into from
Closed

Continuous Profiling Implementation #282

wants to merge 10 commits into from

Conversation

@santigimeno
Copy link
Member

@santigimeno santigimeno commented Mar 25, 2025

Continuous Profiling Implementation

This PR implements continuous profiling capabilities in N|Solid, allowing automatic collection of CPU profiles at configurable intervals.

Key Changes

  • Core Implementation:

    • Created a new ContinuousProfiler class to manage automatic profile collection
    • Hardened NSolidCPUProfiler with improved stability checks and shutdown handling
    • Integrated continuous profiling with the environment list management system

  • Configuration Options:

    • Added environment variables:
      • NSOLID_CONT_CPU_PROFILE - Enable/disable continuous CPU profiling
      • NSOLID_CONT_CPU_PROFILE_INTERVAL - Set interval between profiles (default: 30s)
    • Added corresponding configuration properties:
      • contCpuProfile
      • contCpuProfileInterval

  • gRPC Agent Support:

    • Added ExportContinuousProfile RPC to the service protocol
    • Enhanced AssetStream to support continuous profile data transmission
    • Implemented conflict resolution between manual and continuous profiling
    • Added proper cleanup in shutdown sequence

  • Testing:

    • Added configuration tests for both API and environment variables
    • Created gRPC continuous profiling tests to verify data transmission and format

Technical Details

  • Profile data is collected using an AsyncTSQueue for improved reliability
  • Timestamps are included with profiles for accurate timeline representation
  • Default profiling interval is set to 30 seconds

@santigimeno santigimeno self-assigned this Mar 25, 2025
@santigimeno santigimeno changed the title Santi/continuous profiling Continuous Profiling Implementation Mar 25, 2025
@santigimeno santigimeno force-pushed the santi/continuous_profiling branch 3 times, most recently from e995322 to 4870740 Compare March 27, 2025 11:27
juanarbol
juanarbol approved these changes Apr 1, 2025
View reviewed changes
Copy link
Contributor

@juanarbol juanarbol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@santigimeno santigimeno force-pushed the santi/fix_flaky_otlp_grpc_metrics_test branch 2 times, most recently from c19d441 to f18e72e Compare April 10, 2025 13:17
@santigimeno santigimeno changed the base branch from santi/fix_flaky_otlp_grpc_metrics_test to node-v22.x-nsolid-v5.x April 10, 2025 13:19
@santigimeno
agents: ProfileCollector to use AsyncTSQueue
ab8b645 
Replace manual message handling with AsyncTSQueue for profile data.
Remove initialize() method as initialization is now handled in constructor.
Simplify profile callback implementation.
Remove process_profiles() as AsyncTSQueue handles this automatically.
Update ZMQ and gRPC agents to remove initialize() calls.
@santigimeno
src: implement ContinuousProfiler class
affcaca
@santigimeno
src: harden NSolidCPUProfiler
e1e4e89 
Add is_running checks to all public methods.
Prevent operations on deleted profiler instances.
Add early return guards to TakeCpuProfile, StopProfiling, and
StopProfilingSync.
Improve stability during shutdown sequence.
@santigimeno
lib: add continuous CPU profiling configuration
759ef7e 
Add NSOLID_CONT_CPU_PROFILE environment variable support.
Add NSOLID_CONT_CPU_PROFILE_INTERVAL environment variable support.
Implement contCpuProfile and contCpuProfileInterval config options.
Set default interval to 30000ms (30 seconds).
Update initializeConfig and updateConfig functions.
@santigimeno
src: integrate ContinuousProfiler with EnvList
2c26282 
Create and initialize ContinuousProfiler in EnvList.
Add configuration handling for continuous profiling options.
Update EnvList to enable/disable profiling based on config.
Add proper cleanup in shutdown sequence.
Expose ContinuousProfiler through GetContinuousProfiler() method.
@santigimeno
test: add continuous profiling configuration tests
3329f55 
Add test-nsolid-config-continuous-profiling.js for API configuration.
Add test-nsolid-config-continuous-profiling-env.js for environment
variables.
@santigimeno
agents: add ExportContinuousProfile rpc
e4fc055 
Add ExportContinuousProfile RPC to nsolid_service.proto.
Add start_ts and end_ts fields to Asset message.
@santigimeno
agents: grow AssetStream to support cont profiling
210c063 
Add AssetStreamRpcType enum to differentiate between RPC types.
Modify AssetStream constructor to accept RPC type parameter.
Update Write method to support both ExportAsset and
ExportContinuousProfile.
Add safety assertion for WritesDone method.
@santigimeno
agents: add continuous profiling to gRPC agent
bc12b59 
Add continuous profiler callback and queue.
Register hook with ContinuousProfiler during agent startup.
Implement got_continuous_profile method for handling profile data.
Add conflict resolution between manual and continuous profiling.
Update profile timestamp handling for better accuracy.
Add proper cleanup in shutdown sequence.
@santigimeno
test: add gRPC continuous profiling tests
e638bb7 
Add test-grpc-continuous-profile.mjs for testing continuous profiling.
Update gRPC agent server test fixture.
Test profile data transmission and format.
Verify continuous profiling configuration options.
Please enter the commit message for your changes. Lines starting.
@santigimeno santigimeno force-pushed the santi/continuous_profiling branch from 4870740 to e638bb7 Compare April 10, 2025 13:20
santigimeno added a commit that referenced this pull request Apr 14, 2025
@santigimeno
agents: make ProfileCollector to use AsyncTSQueue
fd05e11 
Replace manual message handling with AsyncTSQueue for profile data.
Remove initialize() method as initialization is now handled in
constructor.
Simplify profile callback implementation.
Remove process_profiles() as AsyncTSQueue handles this automatically.
Update ZMQ and gRPC agents to remove initialize() calls.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Apr 14, 2025
@santigimeno
src: implement ContinuousProfiler class
2d6cee6 
PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Apr 14, 2025
@santigimeno
src: harden NSolidCPUProfiler
7943b5c 
Add is_running checks to all public methods.
Prevent operations on deleted profiler instances.
Add early return guards to TakeCpuProfile, StopProfiling, and
StopProfilingSync.
Improve stability during shutdown sequence.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Apr 14, 2025
@santigimeno
lib: add continuous CPU profiling configuration
ead9239 
Add NSOLID_CONT_CPU_PROFILE environment variable support.
Add NSOLID_CONT_CPU_PROFILE_INTERVAL environment variable support.
Implement contCpuProfile and contCpuProfileInterval config options.
Set default interval to 30000ms (30 seconds).
Update initializeConfig and updateConfig functions.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Apr 14, 2025
@santigimeno
src: integrate ContinuousProfiler with EnvList
d84c59c 
Create and initialize ContinuousProfiler in EnvList.
Add configuration handling for continuous profiling options.
Update EnvList to enable/disable profiling based on config.
Add proper cleanup in shutdown sequence.
Expose ContinuousProfiler through GetContinuousProfiler() method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Apr 14, 2025
@santigimeno
test: add continuous profiling configuration tests
b89c4b4 
Add test-nsolid-config-continuous-profiling.js for API configuration.
Add test-nsolid-config-continuous-profiling-env.js for environment
variables.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Apr 14, 2025
@santigimeno
agents: add ExportContinuousProfile rpc
a339b0d 
Add ExportContinuousProfile RPC to nsolid_service.proto.
Add start_ts and end_ts fields to Asset message.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Apr 14, 2025
@santigimeno
agents: grow AssetStream to support cont profiling
5f1f111 
Add AssetStreamRpcType enum to differentiate between RPC types.
Modify AssetStream constructor to accept RPC type parameter.
Update Write method to support both ExportAsset and
ExportContinuousProfile.
Add safety assertion for WritesDone method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 12, 2025
@santigimeno
src: integrate ContinuousProfiler with EnvList
1ac895b 
Create and initialize ContinuousProfiler in EnvList.
Add configuration handling for continuous profiling options.
Update EnvList to enable/disable profiling based on config.
Add proper cleanup in shutdown sequence.
Expose ContinuousProfiler through GetContinuousProfiler() method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 12, 2025
@santigimeno
test: add continuous profiling configuration tests
62a40a6 
Add test-nsolid-config-continuous-profiling.js for API configuration.
Add test-nsolid-config-continuous-profiling-env.js for environment
variables.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 12, 2025
@santigimeno
agents: add ExportContinuousProfile rpc
6f4153c 
Add ExportContinuousProfile RPC to nsolid_service.proto.
Add start_ts and end_ts fields to Asset message.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 12, 2025
@santigimeno
agents: grow AssetStream to support cont profiling
4488fb5 
Add AssetStreamRpcType enum to differentiate between RPC types.
Modify AssetStream constructor to accept RPC type parameter.
Update Write method to support both ExportAsset and
ExportContinuousProfile.
Add safety assertion for WritesDone method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 12, 2025
@santigimeno
agents: add continuous profiling to gRPC agent
6566a01 
Add continuous profiler callback and queue.
Register hook with ContinuousProfiler during agent startup.
Implement got_continuous_profile method for handling profile data.
Add conflict resolution between manual and continuous profiling.
Update profile timestamp handling for better accuracy.
Add proper cleanup in shutdown sequence.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 12, 2025
@santigimeno
test: add gRPC continuous profiling tests
24fe19d 
Add test-grpc-continuous-profile.mjs for testing continuous profiling.
Update gRPC agent server test fixture.
Test profile data transmission and format.
Verify continuous profiling configuration options.
Please enter the commit message for your changes. Lines starting.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 15, 2025
@santigimeno
src: implement ContinuousProfiler class
9040948 
PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 15, 2025
@santigimeno
lib: add continuous CPU profiling configuration
4cda177 
Add NSOLID_CONT_CPU_PROFILE environment variable support.
Add NSOLID_CONT_CPU_PROFILE_INTERVAL environment variable support.
Implement contCpuProfile and contCpuProfileInterval config options.
Set default interval to 30000ms (30 seconds).
Update initializeConfig and updateConfig functions.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 15, 2025
@santigimeno
src: integrate ContinuousProfiler with EnvList
ddc7c6b 
Create and initialize ContinuousProfiler in EnvList.
Add configuration handling for continuous profiling options.
Update EnvList to enable/disable profiling based on config.
Add proper cleanup in shutdown sequence.
Expose ContinuousProfiler through GetContinuousProfiler() method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 15, 2025
@santigimeno
test: add continuous profiling configuration tests
c08fd67 
Add test-nsolid-config-continuous-profiling.js for API configuration.
Add test-nsolid-config-continuous-profiling-env.js for environment
variables.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 15, 2025
@santigimeno
agents: add ExportContinuousProfile rpc
2462119 
Add ExportContinuousProfile RPC to nsolid_service.proto.
Add start_ts and end_ts fields to Asset message.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 15, 2025
@santigimeno
agents: grow AssetStream to support cont profiling
8de82b1 
Add AssetStreamRpcType enum to differentiate between RPC types.
Modify AssetStream constructor to accept RPC type parameter.
Update Write method to support both ExportAsset and
ExportContinuousProfile.
Add safety assertion for WritesDone method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 15, 2025
@santigimeno
agents: add continuous profiling to gRPC agent
4bb1a08 
Add continuous profiler callback and queue.
Register hook with ContinuousProfiler during agent startup.
Implement got_continuous_profile method for handling profile data.
Add conflict resolution between manual and continuous profiling.
Update profile timestamp handling for better accuracy.
Add proper cleanup in shutdown sequence.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request May 15, 2025
@santigimeno
test: add gRPC continuous profiling tests
dab3bc6 
Add test-grpc-continuous-profile.mjs for testing continuous profiling.
Update gRPC agent server test fixture.
Test profile data transmission and format.
Verify continuous profiling configuration options.
Please enter the commit message for your changes. Lines starting.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 25, 2025
@santigimeno
src: implement ContinuousProfiler class
95694f4 
PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 25, 2025
@santigimeno
lib: add continuous CPU profiling configuration
89bbd13 
Add NSOLID_CONT_CPU_PROFILE environment variable support.
Add NSOLID_CONT_CPU_PROFILE_INTERVAL environment variable support.
Implement contCpuProfile and contCpuProfileInterval config options.
Set default interval to 30000ms (30 seconds).
Update initializeConfig and updateConfig functions.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 25, 2025
@santigimeno
src: integrate ContinuousProfiler with EnvList
8b946c7 
Create and initialize ContinuousProfiler in EnvList.
Add configuration handling for continuous profiling options.
Update EnvList to enable/disable profiling based on config.
Add proper cleanup in shutdown sequence.
Expose ContinuousProfiler through GetContinuousProfiler() method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 25, 2025
@santigimeno
test: add continuous profiling configuration tests
dbc3bfb 
Add test-nsolid-config-continuous-profiling.js for API configuration.
Add test-nsolid-config-continuous-profiling-env.js for environment
variables.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 25, 2025
@santigimeno
agents: add ExportContinuousProfile rpc
6edac0b 
Add ExportContinuousProfile RPC to nsolid_service.proto.
Add start_ts and end_ts fields to Asset message.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 25, 2025
@santigimeno
agents: grow AssetStream to support cont profiling
b152771 
Add AssetStreamRpcType enum to differentiate between RPC types.
Modify AssetStream constructor to accept RPC type parameter.
Update Write method to support both ExportAsset and
ExportContinuousProfile.
Add safety assertion for WritesDone method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 25, 2025
@santigimeno
agents: add continuous profiling to gRPC agent
719d1d9 
Add continuous profiler callback and queue.
Register hook with ContinuousProfiler during agent startup.
Implement got_continuous_profile method for handling profile data.
Add conflict resolution between manual and continuous profiling.
Update profile timestamp handling for better accuracy.
Add proper cleanup in shutdown sequence.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 25, 2025
@santigimeno
test: add gRPC continuous profiling tests
20fcee1 
Add test-grpc-continuous-profile.mjs for testing continuous profiling.
Update gRPC agent server test fixture.
Test profile data transmission and format.
Verify continuous profiling configuration options.
Please enter the commit message for your changes. Lines starting.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
santigimeno added a commit that referenced this pull request Aug 26, 2025
@santigimeno
src: implement ContinuousProfiler class
2d8d138 
PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
PR-URL: #359
Reviewed-By: Rafael Gonzaga <[email protected]>
santigimeno added a commit that referenced this pull request Aug 26, 2025
@santigimeno
lib: add continuous CPU profiling configuration
c870cae 
Add NSOLID_CONT_CPU_PROFILE environment variable support.
Add NSOLID_CONT_CPU_PROFILE_INTERVAL environment variable support.
Implement contCpuProfile and contCpuProfileInterval config options.
Set default interval to 30000ms (30 seconds).
Update initializeConfig and updateConfig functions.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
PR-URL: #359
Reviewed-By: Rafael Gonzaga <[email protected]>
santigimeno added a commit that referenced this pull request Aug 26, 2025
@santigimeno
src: integrate ContinuousProfiler with EnvList
74c6a1e 
Create and initialize ContinuousProfiler in EnvList.
Add configuration handling for continuous profiling options.
Update EnvList to enable/disable profiling based on config.
Add proper cleanup in shutdown sequence.
Expose ContinuousProfiler through GetContinuousProfiler() method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
PR-URL: #359
Reviewed-By: Rafael Gonzaga <[email protected]>
santigimeno added a commit that referenced this pull request Aug 26, 2025
@santigimeno
test: add continuous profiling configuration tests
f1d9a0c 
Add test-nsolid-config-continuous-profiling.js for API configuration.
Add test-nsolid-config-continuous-profiling-env.js for environment
variables.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
PR-URL: #359
Reviewed-By: Rafael Gonzaga <[email protected]>
santigimeno added a commit that referenced this pull request Aug 26, 2025
@santigimeno
agents: add ExportContinuousProfile rpc
bccb1f9 
Add ExportContinuousProfile RPC to nsolid_service.proto.
Add start_ts and end_ts fields to Asset message.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
PR-URL: #359
Reviewed-By: Rafael Gonzaga <[email protected]>
santigimeno added a commit that referenced this pull request Aug 26, 2025
@santigimeno
agents: grow AssetStream to support cont profiling
c104e14 
Add AssetStreamRpcType enum to differentiate between RPC types.
Modify AssetStream constructor to accept RPC type parameter.
Update Write method to support both ExportAsset and
ExportContinuousProfile.
Add safety assertion for WritesDone method.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
PR-URL: #359
Reviewed-By: Rafael Gonzaga <[email protected]>
santigimeno added a commit that referenced this pull request Aug 26, 2025
@santigimeno
agents: add continuous profiling to gRPC agent
4e0c439 
Add continuous profiler callback and queue.
Register hook with ContinuousProfiler during agent startup.
Implement got_continuous_profile method for handling profile data.
Add conflict resolution between manual and continuous profiling.
Update profile timestamp handling for better accuracy.
Add proper cleanup in shutdown sequence.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
PR-URL: #359
Reviewed-By: Rafael Gonzaga <[email protected]>
santigimeno added a commit that referenced this pull request Aug 26, 2025
@santigimeno
test: add gRPC continuous profiling tests
50f9000 
Add test-grpc-continuous-profile.mjs for testing continuous profiling.
Update gRPC agent server test fixture.
Test profile data transmission and format.
Verify continuous profiling configuration options.
Please enter the commit message for your changes. Lines starting.

PR-URL: #282
Reviewed-By: Juan José Arboleda <[email protected]>
PR-URL: #359
Reviewed-By: Rafael Gonzaga <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@juanarbol juanarbol juanarbol approved these changes

@RafaelGSS RafaelGSS Awaiting requested review from RafaelGSS

Assignees

@santigimeno santigimeno

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@santigimeno @juanarbol