This repository was archived by the owner on Nov 15, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
service: storage monitor added #13082
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
34577eb
service: storage monitor added
michalkucharczyk b5f970b
Merge remote-tracking branch 'origin/master' into mku-storage-monitor
michalkucharczyk 37bbba4
Cargo.lock updated
michalkucharczyk ba0f28f
misspell
michalkucharczyk e360685
fs events throttling added
michalkucharczyk fd5679f
minor updates
michalkucharczyk 217234f
filter out non mutating events
michalkucharczyk 8b3310b
misspell
michalkucharczyk 931339c
".git/.scripts/commands/fmt/fmt.sh"
c200159
Update client/service/src/storage_monitor.rs
michalkucharczyk f5501f2
storage-monitor crate added
michalkucharczyk 45171dc
cleanup: configuration + service builder
michalkucharczyk f346394
storage_monitor in custom service (wip)
michalkucharczyk eb49400
copy-paste bad desc fixed
michalkucharczyk 8619766
notify removed
michalkucharczyk a0cd44d
storage_monitor added to node
michalkucharczyk d3f143c
Merge remote-tracking branch 'origin/master' into mku-storage-monitor
a79af9a
fix for clippy
michalkucharczyk 00e7264
publish = false
michalkucharczyk 6c3eb1c
Update bin/node/cli/src/command.rs
michalkucharczyk 58dcf55
Apply suggestions from code review
michalkucharczyk 95b576b
crate name: storage-monitor -> sc-storage-monitor
michalkucharczyk 0b8697e
error handling improved
michalkucharczyk c0fc2ca
Merge remote-tracking branch 'origin/master' into mku-storage-monitor
michalkucharczyk de35dda
Apply suggestions from code review
michalkucharczyk 4e32f3d
publish=false removed
michalkucharczyk 0578588
Merge remote-tracking branch 'origin/master' into mku-storage-monitor
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| [package] | ||
| name = "sc-storage-monitor" | ||
| version = "0.1.0" | ||
| authors = ["Parity Technologies <[email protected]>"] | ||
| edition = "2021" | ||
| license = "GPL-3.0-or-later WITH Classpath-exception-2.0" | ||
| repository = "https://github.com/paritytech/substrate" | ||
| description = "Storage monitor service for substrate" | ||
| homepage = "https://substrate.io" | ||
|
|
||
| [dependencies] | ||
| clap = { version = "4.0.9", features = ["derive", "string"] } | ||
| futures = "0.3.21" | ||
| log = "0.4.17" | ||
| nix = { version = "0.26.1", features = ["fs"] } | ||
| sc-client-db = { version = "0.10.0-dev", default-features = false, path = "../db" } | ||
| sc-utils = { version = "4.0.0-dev", path = "../utils" } | ||
| sp-core = { version = "7.0.0", path = "../../primitives/core" } | ||
| tokio = "1.22.0" | ||
| thiserror = "1.0.30" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| // This file is part of Substrate. | ||
|
|
||
| // Copyright (C) 2022 Parity Technologies (UK) Ltd. | ||
| // SPDX-License-Identifier: GPL-3.0-or-later WITH Classpath-exception-2.0 | ||
|
|
||
| // This program is free software: you can redistribute it and/or modify | ||
| // it under the terms of the GNU General Public License as published by | ||
| // the Free Software Foundation, either version 3 of the License, or | ||
| // (at your option) any later version. | ||
|
|
||
| // This program is distributed in the hope that it will be useful, | ||
| // but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| // GNU General Public License for more details. | ||
|
|
||
| // You should have received a copy of the GNU General Public License | ||
| // along with this program. If not, see <https://www.gnu.org/licenses/>. | ||
|
|
||
| use clap::Args; | ||
| use nix::{errno::Errno, sys::statvfs::statvfs}; | ||
| use sc_client_db::DatabaseSource; | ||
| use sp_core::traits::SpawnEssentialNamed; | ||
| use std::{ | ||
| path::{Path, PathBuf}, | ||
| time::Duration, | ||
| }; | ||
|
|
||
| const LOG_TARGET: &str = "storage-monitor"; | ||
|
|
||
| /// Error type used in this crate. | ||
| #[derive(Debug, thiserror::Error)] | ||
| pub enum Error { | ||
| #[error("IO Error")] | ||
| IOError(#[from] Errno), | ||
| #[error("Out of storage space: available {0}MB, required {1}MB")] | ||
| StorageOutOfSpace(u64, u64), | ||
| } | ||
|
|
||
| /// Parameters used to create the storage monitor. | ||
| #[derive(Default, Debug, Clone, Args)] | ||
| pub struct StorageMonitorParams { | ||
| /// Required available space on database storage. If available space for DB storage drops below | ||
| /// the given threshold, node will be gracefully terminated. If `0` is given monitoring will be | ||
| /// disabled. | ||
| #[arg(long = "db-storage-threshold", value_name = "MB", default_value_t = 1000)] | ||
| pub threshold: u64, | ||
|
|
||
| /// How often available space is polled. | ||
| #[arg(long = "db-storage-polling-period", value_name = "SECONDS", default_value_t = 5, value_parser = clap::value_parser!(u32).range(1..))] | ||
| pub polling_period: u32, | ||
| } | ||
|
|
||
| /// Storage monitor service: checks the available space for the filesystem for fiven path. | ||
| pub struct StorageMonitorService { | ||
| /// watched path | ||
| path: PathBuf, | ||
| /// number of megabytes that shall be free on the filesystem for watched path | ||
| threshold: u64, | ||
| /// storage space polling period (seconds) | ||
| polling_period: u32, | ||
| } | ||
|
|
||
| impl StorageMonitorService { | ||
| /// Creates new StorageMonitorService for given client config | ||
| pub fn try_spawn( | ||
| parameters: StorageMonitorParams, | ||
| database: DatabaseSource, | ||
| spawner: &impl SpawnEssentialNamed, | ||
| ) -> Result<(), Error> { | ||
| Ok(match (parameters.threshold, database.path()) { | ||
| (0, _) => { | ||
| log::info!( | ||
| target: LOG_TARGET, | ||
| "StorageMonitorService: threshold `0` given, storage monitoring disabled", | ||
| ); | ||
| }, | ||
| (_, None) => { | ||
| log::warn!( | ||
| target: LOG_TARGET, | ||
| "StorageMonitorService: no database path to observe", | ||
| ); | ||
| }, | ||
| (threshold, Some(path)) => { | ||
| log::debug!( | ||
| target: LOG_TARGET, | ||
| "Initializing StorageMonitorService for db path: {:?}", | ||
| path, | ||
| ); | ||
|
|
||
| Self::check_free_space(&path, threshold)?; | ||
|
|
||
| let storage_monitor_service = StorageMonitorService { | ||
| path: path.to_path_buf(), | ||
| threshold, | ||
| polling_period: parameters.polling_period, | ||
| }; | ||
|
|
||
| spawner.spawn_essential( | ||
| "storage-monitor", | ||
| None, | ||
| Box::pin(storage_monitor_service.run()), | ||
| ); | ||
| }, | ||
| }) | ||
| } | ||
|
|
||
| /// Main monitoring loop, intended to be spawned as essential task. Quits if free space drop | ||
| /// below threshold. | ||
| async fn run(self) { | ||
| loop { | ||
| tokio::time::sleep(Duration::from_secs(self.polling_period.into())).await; | ||
| if Self::check_free_space(&self.path, self.threshold).is_err() { | ||
| break | ||
| }; | ||
| } | ||
| } | ||
|
|
||
| /// Returns free space in MB, or error if statvfs failed. | ||
| fn free_space(path: &Path) -> Result<u64, Error> { | ||
| statvfs(path) | ||
| .map(|stats| stats.blocks_available() * stats.block_size() / 1_000_000) | ||
michalkucharczyk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| .map_err(Error::from) | ||
| } | ||
|
|
||
| /// Checks if the amount of free space for given `path` is above given `threshold`. | ||
| /// If it dropped below, error is returned. | ||
| /// System errors are silently ignored. | ||
| fn check_free_space(path: &Path, threshold: u64) -> Result<(), Error> { | ||
| match StorageMonitorService::free_space(path) { | ||
| Ok(available_space) => { | ||
| log::trace!( | ||
| target: LOG_TARGET, | ||
| "free: {available_space} , threshold: {threshold}.", | ||
| ); | ||
|
|
||
| if available_space < threshold { | ||
| log::error!(target: LOG_TARGET, "Available space {available_space}MB for path `{}` dropped below threshold: {threshold}MB , terminating...", path.display()); | ||
| Err(Error::StorageOutOfSpace(available_space, threshold)) | ||
| } else { | ||
| Ok(()) | ||
| } | ||
| }, | ||
| Err(e) => { | ||
| log::error!(target: LOG_TARGET, "Could not read available space: {:?}.", e); | ||
| Err(e) | ||
| }, | ||
| } | ||
| } | ||
| } | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.