Skip to content

HDDS-9874. Introduce Metrics for listKeys Dashboard #5745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tanvipenumudy merged 32 commits into from
Jul 8, 2024
Merged

HDDS-9874. Introduce Metrics for listKeys Dashboard #5745

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
c38ead2
added ops per sec and average pagination metric for listkeys
Dec 7, 2023
36ca4fa
updated test cases
Dec 12, 2023
1818a49
Updated all test cases
Dec 13, 2023
f731a19
refactored code
Dec 13, 2023
6d3d979
refactored code
Dec 13, 2023
18e1e00
fixed errors
Dec 14, 2023
6744486
fixed import errors
Dec 14, 2023
a8c1c0a
Merge branch 'master' into HDDS-9874
muskan1012 Dec 18, 2023
3375fce
rearranged imports and refactored code
muskan1012 Dec 18, 2023
68b8238
Merge branch 'master' into HDDS-9874
muskan1012 Jan 4, 2024
e879fc9
removed duplicate code
muskan1012 Jan 8, 2024
94545b6
code change to avoid build failure
muskan1012 Jan 10, 2024
860247b
Merge branch 'master' into HDDS-9874
muskan1012 Jan 28, 2024
3c15956
fixed test cases
muskan1012 Feb 7, 2024
2a3a512
Merge branch 'HDDS-9874' of github.com:muskan1012/ozone into HDDS-9874
muskan1012 Feb 7, 2024
7fb39c6
Merge branch 'apache:master' into HDDS-9874
muskan1012 Feb 7, 2024
1bd6fd7
fixed checkstyle and other issues
muskan1012 Feb 7, 2024
31f8a3c
checkstyle issue resolved
muskan1012 Feb 7, 2024
3a73859
unit test resolution
muskan1012 Feb 7, 2024
6bf2a5e
unit test case failures resolutions
muskan1012 Feb 8, 2024
8671712
listKeysReadFromRocksDbLatency metric added
muskan1012 Feb 8, 2024
a0397eb
removed irrelevant code
muskan1012 Feb 8, 2024
dfd427b
refactored code to avoid null pointer exception
muskan1012 Feb 8, 2024
cf5f43d
Merge branch 'master' into HDDS-9874
muskan1012 Mar 26, 2024
570bc36
Merge branch 'master' into HDDS-9874
muskan1012 Jun 18, 2024
364de9b
updated code to avoid npe
muskan1012 Jun 18, 2024
d26e986
npe resolution for integration-recon
muskan1012 Jun 18, 2024
2bb0cda
refactored code as per review comments
muskan1012 Jun 24, 2024
bf9d575
minor nits fixed
muskan1012 Jun 25, 2024
b0a5832
changes as per the review comment
muskan1012 Jul 1, 2024
1c83093
Apply suggestions from code review
muskan1012 Jul 1, 2024
8139ae7
changed naming convention as per the review comment
muskan1012 Jul 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions ...op-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,11 @@ public static void unregister() {
@Metric(about = "Ratis local command execution latency in nano seconds")
private MutableRate validateAndUpdateCacneLatencyNs;

@Metric(about = "average pagination for listKeys")
private MutableRate listKeysAveragePagination;

@Metric(about = "ops per second for listKeys")
private MutableRate listKeysOpsPerSec;
Copy link
Contributor

@xichen01 xichen01 Dec 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the Latency metric listKeysLatencyNs, maybe we just need to add a cumulative number of listKeys metric(such as: listKeysSize), which we can calculate to get listKeysOpsPerSec and listKeysAveragePagination.

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java

Lines 92 to 93 in 0e91b2a

@Metric(about = "listKeys latency in nanoseconds")
private MutableRate listKeysLatencyNs;

Copy link
Contributor Author

@muskan1012 muskan1012 Jul 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented the cumulative number of listKeys using keyCount, which is set to maxKeys if truncated or cacheKeyMap.size() otherwise.
listKeysAveragePagination is set using keyCount.
listKeysOpsPerSec is calculated as keyCount / ((Time.monotonicNowNanos() - startNanos) / 1_000_000_000.0f).
Added latency tracking with addListKeysReadFromRocksDbLatencyNs.
This approach captures the metrics as you suggested.


public void addLookupLatency(long latencyInNs) {
lookupLatencyNs.add(latencyInNs);
Expand Down Expand Up @@ -210,4 +215,12 @@ public MutableRate getCreateOmResponseLatencyNs() {
public MutableRate getValidateAndUpdateCacneLatencyNs() {
return validateAndUpdateCacneLatencyNs;
}

public void setListKeysAveragePagination(long averagePagination) {
listKeysAveragePagination.add(averagePagination);
}

public void setListKeysOpsPerSec(long opsPerSec) {
listKeysOpsPerSec.add(opsPerSec);
}
}
19 changes: 18 additions & 1 deletion ...p-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,7 @@ public class OmMetadataManagerImpl implements OMMetadataManager,
private final Map<String, TableCacheMetrics> tableCacheMetricsMap =
new HashMap<>();
private SnapshotChainManager snapshotChainManager;
private final OMPerformanceMetrics perfMetrics;

/**
* OmMetadataManagerImpl constructor.
Expand All @@ -334,6 +335,7 @@ public OmMetadataManagerImpl(OzoneConfiguration conf,
OMConfigKeys.OZONE_OM_RATIS_ENABLE_KEY,
OMConfigKeys.OZONE_OM_RATIS_ENABLE_DEFAULT);
this.omEpoch = OmUtils.getOMEpoch(isRatisEnabled);
this.perfMetrics = ozoneManager.getPerfMetrics();
// For test purpose only
ignorePipelineinKey = conf.getBoolean(
"ozone.om.ignore.pipeline", Boolean.TRUE);
Expand All @@ -347,6 +349,7 @@ protected OmMetadataManagerImpl() {
OzoneConfiguration conf = new OzoneConfiguration();
this.lock = new OzoneManagerLock(conf);
this.omEpoch = 0;
perfMetrics = null;
}

public static OmMetadataManagerImpl createCheckpointMetadataManager(
Expand Down Expand Up @@ -381,6 +384,7 @@ private OmMetadataManagerImpl(OzoneConfiguration conf, File dir, String name)
setStore(loadDB(conf, dir, name, true,
java.util.Optional.of(Boolean.TRUE), Optional.empty()));
initializeOmTables(CacheType.PARTIAL_CACHE, false);
perfMetrics = null;
}


Expand Down Expand Up @@ -418,6 +422,7 @@ private OmMetadataManagerImpl(OzoneConfiguration conf, File dir, String name)
stop();
throw e;
}
perfMetrics = null;
}

@Override
Expand Down Expand Up @@ -1177,7 +1182,7 @@ public List<OmBucketInfo> listBuckets(final String volumeName,
public ListKeysResult listKeys(String volumeName, String bucketName,
String startKey, String keyPrefix, int maxKeys)
throws IOException {

long startNanos = Time.monotonicNowNanos();
List<OmKeyInfo> result = new ArrayList<>();
if (maxKeys <= 0) {
return new ListKeysResult(result, false);
Expand Down Expand Up @@ -1276,6 +1281,18 @@ public ListKeysResult listKeys(String volumeName, String bucketName,
}

boolean isTruncated = cacheKeyMap.size() > maxKeys;
long averagePagination;
if(isTruncated)
{
averagePagination = maxKeys;
}
else
{
averagePagination = cacheKeyMap.size();
}
perfMetrics.setListKeysAveragePagination(averagePagination);
long opsPerSec = averagePagination/(Time.monotonicNowNanos() - startNanos);
perfMetrics.setListKeysOpsPerSec(opsPerSec);

// Finally DB entries and cache entries are merged, then return the count
// of maxKeys from the sorted map.
Expand Down