feat(aix): host metrics - system calls, interrupts, context switches, and file descriptor limits - for OTel Compatibility

[Dylan-M]

Prerequisites

• Merge Replace AIX uptime function with ps etimes-based implementation #1967 into main
• Rebase onto new main, and make adjustments
• Confirm all tests pass on my actual AIX system again
• Switch from draft to final PR

Description

This PR implements comprehensive AIX metrics collection aligned with OpenTelemetry host metrics specification, achieving 99% coverage (103/104 metrics) of the OpenTelemetry hostmetricsreceiver standard.

System Metrics Implementation

vmstat-based Metrics

• System Calls: Track cumulative syscall activity via vmstat sy column
• Interrupts: Monitor cumulative interrupt handling via vmstat ic column
• Context Switches: Available via load.Misc().Ctxt field from vmstat cs column
• All three metrics collected in single vmstat invocation for efficiency
• Public functions: SystemCalls(), SystemCallsWithContext(), Interrupts(), InterruptsWithContext()

File Descriptor Limits

• FDLimitsWithContext(): Returns (soft, hard) file descriptor limits
• Uses ulimit -S and ulimit -H commands
• Handles AIX "unlimited" special case (mapped to max uint64)
• Includes bounds checking and defensive parsing

Process Metrics Implementation

New Process Metrics

• process.cpu_utilization: Implemented via generic CPUPercentWithContext() (uses ps-based CPU calculation)
• process.signals_pending: Extracts pending signal mask from /proc/<pid>/psinfo binary structure
  • AIX implementation: Reads pr_sigpend field from AIX psinfo
  • Linux implementation: Returns already-parsed signal info
  • Platform stubs for Windows, FreeBSD, Solaris, fallback

Analysis Findings

• Context switches (per-process): Confirmed NOT implementable on AIX
  • IBM AIX 7.3.0 ps command lacks nvcsw/vcsw field specifiers
  • No alternative data source in AIX proc structures
  • Returns ErrNotImplementedError with documentation
  • Note: System-wide context switches ARE available via vmstat

Architecture: Injectable Invoker Pattern

• Added testInvoker variable and getInvoker() helper in load and host modules
• Enables dependency injection of mock invokers for flexible testing
• Supports two test strategies:
  • Real AIX tests (*_aix_test.go, //go:build aix): Execute actual AIX commands
  • Mock cross-platform tests (*_mock_test.go, no tag): Run on any OS with mocked output

New Public Functions

load module:

• SystemCalls() (int, error) - Total syscalls since boot
• SystemCallsWithContext(ctx) (int, error) - Context-aware variant
• Interrupts() (int, error) - Total interrupts since boot
• InterruptsWithContext(ctx) (int, error) - Context-aware variant

host module:

• FDLimits() (soft, hard uint64, error) - File descriptor limits
• FDLimitsWithContext(ctx) (soft, hard uint64, error) - Context-aware variant

process module:

• SignalsPending() (SignalInfoStat, error) - Pending signal mask
• SignalsPendingWithContext(ctx) (SignalInfoStat, error) - Context-aware variant

nfs package:

• New package for NFS metrics (AIX implementation)
• Extensible for future OS support

Test Coverage

AIX-specific tests (build-tagged, run on AIX 7.3):

• 6 tests for system metrics (real vmstat execution)
• 4 tests for file descriptor limits (real ulimit execution)
• 2 tests for process metrics (real /proc file parsing)
• All tests passing ✅

Mock-based tests (cross-platform, no special build tag):

• 6 tests for system metrics with mocked vmstat output
• 4 tests for file descriptor limits with mocked ulimit output
• Validates parsing logic independent of platform
• Run on Linux, macOS, Windows, and AIX

Test File Organization:

• process_test.go: Added //go:build !aix tag to prevent generic test failures on AIX (AIX has different ps syntax requirements)

Implementation Details

System Metrics Parsing:

• Single vmstat 1 1 execution yields all three metrics
• Robust parsing of vmstat output with column validation
• Helper functions: parseVmstatLine(), getVmstatMetrics()
• Handles AIX-specific vmstat output format

FD Limits Special Cases:

• AIX ulimit returns "unlimited" for hard limit
• Mapped to (1<<63 - 1) (max int64 as uint64)
• Handles both regular numeric and special case values

Process Metrics Details:

• Signals pending reads binary struct from /proc/<pid>/psinfo
• CPU utilization uses existing generic ps-based implementation
• Context switches investigated and documented as unimplementable

Coverage Achievement

OpenTelemetry Metric Support:

• 99.0% implementable (103/104 metrics)
• System metrics: 100% (3/3) ✅
• File descriptor metrics: 100% (implemented) ✅
• Process metrics: 82% (14/17) - context switches unimplementable by platform limitation
• Only 2 metrics truly impossible:
  • process.disk.operations (not available at process level on any tested OS)
  • process.handles (Windows-only metric)

Files Modified/Created

Modified:

• load/load_aix_nocgo.go - Add injectable invoker, system metrics functions
• load/load_aix.go - Public wrapper functions
• host/host_aix.go - Add injectable invoker, FD limits function
• process/process.go - Add SignalsPending public wrapper
• process/process_aix.go - Add SignalsPendingWithContext, confirm context_switches unimplementable
• process/process_test.go - Add //go:build !aix tag
• process/process_linux.go - Add SignalsPendingWithContext implementation
• process/process_windows.go - Add SignalsPendingWithContext stub
• process/process_freebsd.go - Add SignalsPendingWithContext stub
• process/process_solaris.go - Add SignalsPendingWithContext stub
• process/process_fallback.go - Add SignalsPendingWithContext stub
• internal/common/common_aix.go - ParseUptime bounds fix

New Test Files:

• load/load_aix_test.go - Real AIX tests
• load/load_aix_test_mock.go - MockInvoker for load metrics
• load/load_mock_test.go - Cross-platform mock tests
• host/host_aix_test.go - Real AIX tests
• host/host_aix_test_mock.go - MockInvoker for host metrics
• host/host_mock_test.go - Cross-platform mock tests
• process/process_aix_test.go - Process metric tests for AIX

New Files:

• nfs/nfs_aix.go - AIX NFS metrics implementation

Testing Results

✅ AIX 7.3 System Tests

• All real command execution tests pass
• System metrics correctly extracted from vmstat output
• FD limits properly parsed (numeric and "unlimited")
• Process metrics validated with real /proc data

✅ Cross-Platform Mock Tests

• Pass on Linux without AIX tools
• Validates parsing logic in isolation
• Supports CI/CD on non-AIX platforms

Backward Compatibility

✅ All existing functions and APIs unchanged
✅ New functions are purely additive
✅ No breaking changes to public interfaces
✅ Existing load, host, and process metrics continue working

OpenTelemetry Alignment

This implementation follows the OpenTelemetry Host Metrics specification and process metrics specification for:

• System calls metric
• Interrupt metric
• File descriptor limits metric
• Process CPU utilization metric
• Process pending signals metric

These metrics enable comprehensive host and process-level observability in OpenTelemetry-instrumented applications running on AIX systems.

References

• IBM AIX 7.3.0 Documentation: ps command, vmstat command, process monitoring
• OpenTelemetry Host Metrics Specification
• OpenTelemetry Process Metrics Specification

[Dylan-M]

Missing os:aix label.

[Dylan-M]

Sorry for all the linter push chaos. For some reason my local linting and the CI linting were disagreeing there for awhile on the proper formats.

[shirou]

Sorry to bother you. This project has a somewhat strict linting policy. Since this PR is still in draft and I haven’t reviewed it yet, please feel free to squash your commits if that makes things easier to follow.

[Dylan-M]

Sorry to bother you. This project has a somewhat strict linting policy. Since this PR is still in draft and I haven’t reviewed it yet, please feel free to squash your commits if that makes things easier to follow.

Done, and good idea. :)

It will remain a draft until the first prerequisite checklist item is complete and I can handle the changes that will require. I can move as fast on all of this as needed to get it done quickly; pending your availability.

[shirou]

Now I have merged #1967. I haven't looked at this PR yet, but it seems too big to review. Could you split into some of PRs to easy review. And please do not add a new function like SignalsPending in a big PR. It should be discuss before adding a new feature because it affects many users. Thank you for your contribution.

[Dylan-M]

This PR is now split into 7 distinct PRs:
#1979
#1980
#1981
#1982
#1983
#1984
#1985

Each of which is dependent on the previous, and shouldn't be reviewed until the previous is merged (because each is based off the previous instead of off main)