Add miss_count and removed_in_bytes fields in /_nodes/stats/file_cache API and fix used_in_bytes field collision #19656

x-INFiN1TY-x · 2025-10-16T11:23:36Z

Summary

This CR enhances the OpenSearch /_nodes/stats/file_cache API by adding comprehensive cache monitoring capabilities and fixes a critical JSON field-name collision that caused nested used_in_bytes to be reported as 0.

miss_count field: Extends cache miss tracking to all nested statistics objects (previously only available at root level)
removed_in_bytes / removed (human-readable): Adds explicit removal tracking to distinguish operational removals from capacity-driven evictions
Bug fix: Introduce PINNED_IN_BYTES and stop writing pinned usage to USED_IN_BYTES (removes field-name collision so nested used_in_bytes now report actual usage)

API Response Enhancement

Before (Incomplete Visibility):

{
  "_nodes": {"total": 1, "successful": 1, "failed": 0},
  "cluster_name": "REDACTED:REDACTED-REDACTED",
  "nodes": {
    "_-REDACTED": {
      "aggregate_file_cache": {
        "active_in_bytes": 0,
        "total_in_bytes": 368353338982,
        "used_in_bytes": 0,
        "pinned_in_bytes": 0,
        "evictions_in_bytes": 0,
        "hit_count": 1008237,
        "miss_count": 904846,              // Only at ROOT level

        "over_all_stats": {
          "active_in_bytes": 0,
          "used_in_bytes": 0,              
          "evictions_in_bytes": 0,
          "hit_count": 1008237
          // ❌ MISSING: miss_count
          // ❌ MISSING: removed_in_bytes
        },
        "full_file_stats": {
          "active_in_bytes": 0,
          "used_in_bytes": 0,
          "evictions_in_bytes": 0,
          "hit_count": 69
          // ❌ MISSING: miss_count
          // ❌ MISSING: removed_in_bytes
        },
        "block_file_stats": {
          "active_in_bytes": 0,
          "used_in_bytes": 0,
          "evictions_in_bytes": 0,
          "hit_count": 1008168
          // ❌ MISSING: miss_count
          // ❌ MISSING: removed_in_bytes
        },
        "pinned_file_stats": {
          "active_in_bytes": 0,
          "used_in_bytes": 0,
          "evictions_in_bytes": 0,
          "hit_count": 0
          // ❌ MISSING: miss_count
          // ❌ MISSING: removed_in_bytes
        }
      }
    }
  }
}

After:

{
  "_nodes": {"total": 1, "successful": 1, "failed": 0},
  "cluster_name": "REDACTED:REDACTED-REDACTED",
  "nodes": {
    "REDACTED": {
      "aggregate_file_cache": {
        "active_in_bytes": 0,
        "active": "0b",
        "total_in_bytes": 368353338982,
        "total": "343.2gb",
        "used_in_bytes": 0,
        "used": "0b",
        "pinned_in_bytes": 0,
        "pinned": "0b",
        "evictions_in_bytes": 0,
        "evictions": "0b",
        "removed_in_bytes": 67890,         // NEW: Total explicit removals
        "removed": "66.3kb",               // NEW: Human readable
        "hit_count": 1008237,
        "miss_count": 904846,

        "over_all_stats": {
          "active_in_bytes": 0,
          "active": "0b",
          "used_in_bytes": 382533541,      
          "used": "364.9mb",
          "evictions_in_bytes": 0,
          "evictions": "0b",
          "removed_in_bytes": 67890,       // NEW: Overall removals
          "removed": "66.3kb",
          "active_percent": 0,
          "hit_count": 1008237,
          "miss_count": 904846             // NEW: Overall miss count
        },
        "full_file_stats": {
          "active_in_bytes": 0,
          "active": "0b",
          "used_in_bytes": 41445300,       
          "used": "39.5mb",
          "evictions_in_bytes": 0,
          "evictions": "0b",
          "removed_in_bytes": 22212,       // NEW
          "removed": "21.7kb",
          "active_percent": 0,
          "hit_count": 69,
          "miss_count": 0                  // NEW
        },
        "block_file_stats": {
          "active_in_bytes": 0,
          "active": "0b",
          "used_in_bytes": 341088241,      
          "used": "325.2mb",
          "evictions_in_bytes": 0,
          "evictions": "0b",
          "removed_in_bytes": 45678,       // NEW
          "removed": "44.6kb",
          "active_percent": 0,
          "hit_count": 1008168,
          "miss_count": 904846             // NEW
        },
        "pinned_file_stats": {
          "active_in_bytes": 0,
          "active": "0b",
          "used_in_bytes": 0,
          "used": "0b",
          "evictions_in_bytes": 0,
          "evictions": "0b",
          "removed_in_bytes": 0,           // NEW
          "removed": "0b",
          "active_percent": 0,
          "hit_count": 0,
          "miss_count": 0                  // NEW
        }
      }
    }
  }
}

github-actions · 2025-10-16T12:30:03Z

❌ Gradle check result for 5fc979c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

x-INFiN1TY-x · 2025-10-16T20:51:29Z

Flaky tests :

github-actions · 2025-10-16T22:35:10Z

❌ Gradle check result for 658c9a5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Gagan6164

Changes looks good to me.

server/src/main/java/org/opensearch/index/store/remote/filecache/FileCacheStats.java

github-actions · 2025-10-17T13:24:23Z

❌ Gradle check result for 2c1bef7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-21T11:07:02Z

❌ Gradle check result for 42c9c8f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2025-11-20T15:23:09Z

This PR is stalled because it has been open for 30 days with no activity.

gbbafna · 2025-11-24T03:53:47Z

@x-INFiN1TY-x : can you rebase this please ?

…s fields Add cache miss tracking and explicit removal tracking to file cache statistics: Add miss_count field to all nested statistics objects for hit ratio analysis Add removed_in_bytes field to distinguish manual removals from automatic evictions Update FileCacheStats with getRemoved() method and enhanced JSON serialization Update AggregateFileCacheStats with aggregate JSON serialization for new fields Update FileCache fileCacheStats() method to pass removal data Add comprehensive test coverage including MissCountValidationTest Update all FileCacheStats constructor calls to include removed parameter Enables hit ratio calculation, cache pressure monitoring, and performance debugging across different file types in the cache hierarchy. Signed-off-by: tanishq ranjan <[email protected]>

Signed-off-by: tanishq ranjan <[email protected]>

x-INFiN1TY-x · 2025-11-24T10:02:39Z

@gbbafna Done!

All FileCacheStats constructor signature issues have been resolved: - Fixed 24 constructor calls across 4 test files - Added version compatibility for V_3_4_0 - All gradle build and test commands pass locally Signed-off-by: Tanishq Ranjan <[email protected]>

github-actions · 2025-11-24T12:11:21Z

❌ Gradle check result for 2677c17: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-11-24T13:25:43Z

❌ Gradle check result for 2677c17: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-11-24T17:43:10Z

❌ Gradle check result for 2677c17: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-11-25T11:51:12Z

✅ Gradle check result for 2677c17: SUCCESS

codecov · 2025-11-25T11:51:47Z

Codecov Report

❌ Patch coverage is 78.94737% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.23%. Comparing base (97d3864) to head (2677c17).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...h/index/store/remote/filecache/FileCacheStats.java	69.23%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19656      +/-   ##
============================================
- Coverage     73.33%   73.23%   -0.11%     
+ Complexity    71679    71659      -20     
============================================
  Files          5790     5790              
  Lines        327549   327565      +16     
  Branches      47181    47183       +2     
============================================
- Hits         240217   239895     -322     
- Misses        68080    68434     +354     
+ Partials      19252    19236      -16

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…e API and fix used_in_bytes field collision (opensearch-project#19656) Signed-off-by: tanishq ranjan <[email protected]>

x-INFiN1TY-x marked this pull request as ready for review October 16, 2025 11:33

x-INFiN1TY-x requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners October 16, 2025 11:33

x-INFiN1TY-x changed the title ~~Enhance /_nodes/stats/file_cache API with miss_count and removed_in_bytes fields~~ Add miss_count and removed_in_bytes fields in /_nodes/stats/file_cache API Oct 16, 2025

x-INFiN1TY-x changed the title ~~Add miss_count and removed_in_bytes fields in /_nodes/stats/file_cache API~~ Add miss_count and removed_in_bytes fields in /_nodes/stats/file_cache API and fix used_in_bytes field collision Oct 16, 2025

Gagan6164 approved these changes Oct 17, 2025

View reviewed changes

ajaymovva suggested changes Oct 17, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/store/remote/filecache/FileCacheStats.java Outdated Show resolved Hide resolved

x-INFiN1TY-x force-pushed the EnhanceNodesStatsFileCacheAPI branch from 658c9a5 to 2c1bef7 Compare October 17, 2025 12:51

opensearch-trigger-bot bot added the stalled Issues that have stalled label Nov 20, 2025

tqranjan added 4 commits November 24, 2025 09:43

Version check added

d752150

Signed-off-by: tanishq ranjan <[email protected]>

rerun for flaky tests

72f4b5d

Signed-off-by: tanishq ranjan <[email protected]>

gradle rerun

9147726

Signed-off-by: tanishq ranjan <[email protected]>

x-INFiN1TY-x force-pushed the EnhanceNodesStatsFileCacheAPI branch from 42c9c8f to 9820c2c Compare November 24, 2025 10:01

x-INFiN1TY-x force-pushed the EnhanceNodesStatsFileCacheAPI branch from 9820c2c to 2677c17 Compare November 24, 2025 10:08

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Nov 24, 2025

This was referenced Nov 24, 2025

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobStoreRepositoryTests #14291

Open

[AUTOCUT] Gradle Check Flaky Test Report for RestoreShallowSnapshotV2IT #16658

Open

gbbafna added the skip-changelog label Nov 27, 2025

gbbafna approved these changes Nov 27, 2025

View reviewed changes

gbbafna merged commit a0a40bd into opensearch-project:main Nov 27, 2025
40 of 48 checks passed

Add miss_count and removed_in_bytes fields in /_nodes/stats/file_cache API and fix used_in_bytes field collision #19656

Add miss_count and removed_in_bytes fields in /_nodes/stats/file_cache API and fix used_in_bytes field collision #19656

Uh oh!

Conversation

x-INFiN1TY-x commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

API Response Enhancement

Before (Incomplete Visibility):

After:

Uh oh!

github-actions bot commented Oct 16, 2025

Uh oh!

x-INFiN1TY-x commented Oct 16, 2025

Uh oh!

github-actions bot commented Oct 16, 2025

Uh oh!

Gagan6164 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 21, 2025

Uh oh!

opensearch-trigger-bot bot commented Nov 20, 2025

Uh oh!

gbbafna commented Nov 24, 2025

Uh oh!

x-INFiN1TY-x commented Nov 24, 2025

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

codecov bot commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

x-INFiN1TY-x commented Oct 16, 2025 •

edited

Loading

codecov bot commented Nov 25, 2025 •

edited

Loading