Skip to content

HDDS-10184. Fix ManagedStatistics not closed properly #6055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 3 commits into from
Jan 25, 2024
Merged

HDDS-10184. Fix ManagedStatistics not closed properly #6055

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dd0e0fa
HDDS-10184. Fix ManagedStatistics not closed properly
Jan 22, 2024
3764acd
change close Statistics scope
Jan 23, 2024
d051057
Statistics closed in RDBStore#close
Jan 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 0 additions & 15 deletions ...rvice/src/main/java/org/apache/hadoop/ozone/container/metadata/AbstractDatanodeStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,14 +32,12 @@
import org.apache.hadoop.hdds.utils.db.TableIterator;
import org.apache.hadoop.hdds.utils.db.managed.ManagedColumnFamilyOptions;
import org.apache.hadoop.hdds.utils.db.managed.ManagedDBOptions;
import org.apache.hadoop.hdds.utils.db.managed.ManagedStatistics;
import org.apache.hadoop.ozone.container.common.helpers.BlockData;
import org.apache.hadoop.ozone.container.common.helpers.ChunkInfoList;
import org.apache.hadoop.ozone.container.common.interfaces.BlockIterator;
import org.apache.hadoop.ozone.container.common.statemachine.DatanodeConfiguration;
import org.apache.hadoop.ozone.container.common.utils.db.DatanodeDBProfile;
import org.rocksdb.InfoLogLevel;
import org.rocksdb.StatsLevel;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand All @@ -49,9 +47,6 @@

import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_DB_PROFILE;
import static org.apache.hadoop.hdds.utils.db.DBStoreBuilder.HDDS_DEFAULT_DB_PROFILE;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS_DEFAULT;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF;

/**
* Implementation of the {@link DatanodeStore} interface that contains
Expand Down Expand Up @@ -113,16 +108,6 @@ public void start(ConfigurationSource config)
options.setMaxTotalWalSize(maxWalSize);
}

String rocksDbStat = config.getTrimmed(
OZONE_METADATA_STORE_ROCKSDB_STATISTICS,
OZONE_METADATA_STORE_ROCKSDB_STATISTICS_DEFAULT);

if (!rocksDbStat.equals(OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF)) {
ManagedStatistics statistics = new ManagedStatistics();
statistics.setStatsLevel(StatsLevel.valueOf(rocksDbStat));
options.setStatistics(statistics);
}

DatanodeConfiguration dc =
config.getObject(DatanodeConfiguration.class);
// Config user log files
Expand Down
17 changes: 9 additions & 8 deletions hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/DBStoreBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,9 @@ public final class DBStoreBuilder {
// The column family options that will be used for any column families
// added by name only (without specifying options).
private ManagedColumnFamilyOptions defaultCfOptions;
// Initialize the Statistics instance if ROCKSDB_STATISTICS enabled
private ManagedStatistics statistics;

private String dbname;
private Path dbPath;
private String dbJmxBeanNameName;
Expand Down Expand Up @@ -188,6 +191,11 @@ private void setDBOptionsProps(ManagedDBOptions dbOptions) {
if (maxNumberOfOpenFiles != null) {
dbOptions.setMaxOpenFiles(maxNumberOfOpenFiles);
}
if (!rocksDbStat.equals(OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF)) {
statistics = new ManagedStatistics();
statistics.setStatsLevel(StatsLevel.valueOf(rocksDbStat));
dbOptions.setStatistics(statistics);
}
Comment on lines +194 to +198
Copy link
Contributor

@adoroszlai adoroszlai Jan 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ManagedStatistics statistics could be a local variable in build().

I don't think it makes any difference functionally, but it would be easier to follow its lifecycle. With instance variable, the reader needs to consider what happens if the same builder is used to build multiple RDBStore instances.

Copy link
Contributor Author

@whbing whbing Jan 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The global variable statistics are easier to pass to the RDBStore In current method structure, , nothing else.

}

/**
Expand Down Expand Up @@ -217,7 +225,7 @@ public DBStore build() throws IOException {
throw new IOException("The DB destination directory should exist.");
}

return new RDBStore(dbFile, rocksDBOption, writeOptions, tableConfigs,
return new RDBStore(dbFile, rocksDBOption, statistics, writeOptions, tableConfigs,
registry.build(), openReadOnly, maxFSSnapshots, dbJmxBeanNameName,
enableCompactionDag, maxDbUpdatesSizeThreshold, createCheckpointDirs,
configuration, threadNamePrefix);
Expand Down Expand Up @@ -413,13 +421,6 @@ protected void log(InfoLogLevel infoLogLevel, String s) {
dbOptions.setWalTtlSeconds(rocksDBConfiguration.getWalTTL());
dbOptions.setWalSizeLimitMB(rocksDBConfiguration.getWalSizeLimit());

// Create statistics.
if (!rocksDbStat.equals(OZONE_METADATA_STORE_ROCKSDB_STATISTICS_OFF)) {
ManagedStatistics statistics = new ManagedStatistics();
statistics.setStatsLevel(StatsLevel.valueOf(rocksDbStat));
dbOptions.setStatistics(statistics);
}

return dbOptions;
}

Expand Down
12 changes: 9 additions & 3 deletions hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.apache.hadoop.hdds.utils.db.RocksDatabase.ColumnFamily;
import org.apache.hadoop.hdds.utils.db.managed.ManagedCompactRangeOptions;
import org.apache.hadoop.hdds.utils.db.managed.ManagedDBOptions;
import org.apache.hadoop.hdds.utils.db.managed.ManagedStatistics;
import org.apache.hadoop.hdds.utils.db.managed.ManagedTransactionLogIterator;
import org.apache.hadoop.hdds.utils.db.managed.ManagedWriteOptions;
import org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer;
Expand Down Expand Up @@ -77,10 +78,11 @@ public class RDBStore implements DBStore {
// number in request to avoid increase in heap memory.
private final long maxDbUpdatesSizeThreshold;
private final ManagedDBOptions dbOptions;
private final ManagedStatistics statistics;
private final String threadNamePrefix;

@SuppressWarnings("parameternumber")
public RDBStore(File dbFile, ManagedDBOptions dbOptions,
public RDBStore(File dbFile, ManagedDBOptions dbOptions, ManagedStatistics statistics,
ManagedWriteOptions writeOptions, Set<TableConfig> families,
CodecRegistry registry, boolean readOnly, int maxFSSnapshots,
String dbJmxBeanName, boolean enableCompactionDag,
Expand All @@ -97,6 +99,7 @@ public RDBStore(File dbFile, ManagedDBOptions dbOptions,
codecRegistry = registry;
dbLocation = dbFile;
this.dbOptions = dbOptions;
this.statistics = statistics;

try {
if (enableCompactionDag) {
Expand All @@ -119,8 +122,8 @@ public RDBStore(File dbFile, ManagedDBOptions dbOptions,
if (dbJmxBeanName == null) {
dbJmxBeanName = dbFile.getName();
}
metrics = RocksDBStoreMetrics.create(dbOptions.statistics(), db,
dbJmxBeanName);
// Use statistics instead of dbOptions.statistics() to avoid repeated init.
metrics = RocksDBStoreMetrics.create(statistics, db, dbJmxBeanName);
if (metrics == null) {
LOG.warn("Metrics registration failed during RocksDB init, " +
"db path :{}", dbJmxBeanName);
Expand Down Expand Up @@ -231,6 +234,9 @@ public void close() throws IOException {
RocksDBCheckpointDifferHolder
.invalidateCacheEntry(rocksDBCheckpointDiffer.getMetadataDir());
}
if (statistics != null) {
IOUtils.close(LOG, statistics);
}
IOUtils.close(LOG, db);
Comment on lines +237 to 240
Copy link
Contributor

@adoroszlai adoroszlai Jan 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: IOUtils.close accepts multiple objects to be closed, and handles null values, too.

Suggested change
if (statistics != null) {
IOUtils.close(LOG, statistics);
}
IOUtils.close(LOG, db);
IOUtils.close(LOG, db, statistics);

}

Expand Down
2 changes: 1 addition & 1 deletion hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ public static RDBStore newRDBStore(File dbFile, ManagedDBOptions options,
Set<TableConfig> families,
long maxDbUpdatesSizeThreshold)
throws IOException {
return new RDBStore(dbFile, options, new ManagedWriteOptions(), families,
return new RDBStore(dbFile, options, null, new ManagedWriteOptions(), families,
CodecRegistry.newBuilder().build(), false, 1000, null, false,
maxDbUpdatesSizeThreshold, true, null, "");
}
Expand Down
2 changes: 2 additions & 0 deletions ...st/src/test/java/org/apache/hadoop/hdds/utils/db/managed/TestRocksObjectLeakDetector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import java.util.concurrent.TimeoutException;
import java.util.function.Supplier;

import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_STORE_ROCKSDB_STATISTICS;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;

Expand All @@ -49,6 +50,7 @@ public class TestRocksObjectLeakDetector {
static void setUp() throws IOException, InterruptedException,
TimeoutException {
OzoneConfiguration conf = new OzoneConfiguration();
conf.set(OZONE_METADATA_STORE_ROCKSDB_STATISTICS, "ALL");
String clusterId = UUID.randomUUID().toString();
String scmId = UUID.randomUUID().toString();
String omServiceId = "omServiceId1";
Expand Down