From 712f41a1835c74bcba427287df6c85be70c1117e Mon Sep 17 00:00:00 2001 From: "Jeffrey T. Palmer" Date: Wed, 20 May 2020 14:34:15 -0400 Subject: [PATCH] Update shredder documentation --- bin/xdmod-shredder | 14 +++++++++----- docs/resource-manager-lsf.md | 8 -------- docs/resource-manager-pbs.md | 21 +++++++++++++++------ docs/resource-manager-sge.md | 8 -------- docs/resource-manager-slurm.md | 8 -------- docs/shredder.md | 13 +++---------- 6 files changed, 27 insertions(+), 45 deletions(-) diff --git a/bin/xdmod-shredder b/bin/xdmod-shredder index a988555c1d..bdfbbff3c0 100755 --- a/bin/xdmod-shredder +++ b/bin/xdmod-shredder @@ -296,11 +296,15 @@ Usage: xdmod-shredder [-v] -r resource -f format [-i file|-d dir] storage data. -d, --dir *directory* - Specify a directory containing log files to shred. Log files in - the directory must be named in YYYYMMDD format where the file - name corresponds to the end date of the jobs it contains. Only - files with dates that are more recent than the end date of the - most recent job will be shredded. + Specify a directory containing log files to shred. + + For PBS format log files, files in the directory must be named in + YYYYMMDD format where the file name corresponds to the end date of the + jobs it contains. Only files with dates that are more recent than the + end date of the most recent job will be shredded. + + For all other formats, all files in the directory will be shredded. + Cloud and storage files must end in ".json". --host-filter *host-regex* Specify the host filter regex for jobs. Any job that does not diff --git a/docs/resource-manager-lsf.md b/docs/resource-manager-lsf.md index f4119e1ee1..3c87ea4d1c 100644 --- a/docs/resource-manager-lsf.md +++ b/docs/resource-manager-lsf.md @@ -17,11 +17,3 @@ PI Data Source Open XDMoD uses the `projectName` field from `lsb.acct` as the default PI data source. - -Unsupported Shredder Features ------------------------------ - -The xdmod-shredder `-d`/`--dir` option was designed to work with the -accounting log naming convention used by PBS/TORQUE. If you are not -using the same convention (files are named `YYYYMMDD` corresponding to -the current date), do not use this option. diff --git a/docs/resource-manager-pbs.md b/docs/resource-manager-pbs.md index d20f25013a..5a2053fb86 100644 --- a/docs/resource-manager-pbs.md +++ b/docs/resource-manager-pbs.md @@ -7,11 +7,20 @@ Open XDMoD supports PBS including OpenPBS and TORQUE. Logs Files ---------- -The xdmod-shredder `-d`/`--dir` option was designed to work with the -accounting log naming convention used by PBS/TORQUE. If you are not -using the same convention (files are named `YYYYMMDD` corresponding to -the current date), do not use this option. These log files are -typically stored in `/var/spool/pbs/server_priv/accounting`, -`$PBSROOT/server_priv/accounting`, +When using the `pbs` format, the `xdmod-shredder` `-d`/`--dir` option expects +log files to use the accounting log naming convention used by PBS/TORQUE. If +you are not using the same convention (files are named `YYYYMMDD` corresponding +to the current date), do not use this option. + +The log file for the current day will be ignored (along with any files +that correspond to future dates). This is intended to prevent the +shredding of partial log files. + +If the database is empty all files that meet the above constraints will +be shredded. If there is data in the database, only files dated after +the date of the most recent job will be shredded. + +These log files are typically stored in +`/var/spool/pbs/server_priv/accounting`, `$PBSROOT/server_priv/accounting`, `/var/spool/torque/server_priv/accounting`, or `$TORQUEROOT/server_priv/accounting`. diff --git a/docs/resource-manager-sge.md b/docs/resource-manager-sge.md index aba10bb3b8..9a846b5cec 100644 --- a/docs/resource-manager-sge.md +++ b/docs/resource-manager-sge.md @@ -32,11 +32,3 @@ data, the unique key constraint on the `shredded_job_sge` table will prevent the insertion of this data, but will instead update the primary key. This will prevent duplicate data from being entered into the database, even though the entire log file will still be parsed. - -Unsupported Shredder Features ------------------------------ - -The xdmod-shredder `-d`/`--dir` option was designed to work with the -accounting log naming convention used by PBS/TORQUE. If you are not -using the same convention (files are named `YYYYMMDD` corresponding to -the current date), do not use this option. diff --git a/docs/resource-manager-slurm.md b/docs/resource-manager-slurm.md index 3aa4309407..2aa10a72ab 100644 --- a/docs/resource-manager-slurm.md +++ b/docs/resource-manager-slurm.md @@ -57,11 +57,3 @@ $ xdmod-shredder -r *cluster* -f slurm -i /tmp/slurm.log prevent ambiguities caused by clock changes due to daylight savings. The shredder will assume input times are in UTC regardless of your system time zone. - -Unsupported Shredder Features ------------------------------ - -The `xdmod-shredder` `-d`/`--dir` option was designed to work with the -accounting log naming convention used by PBS/TORQUE. If you are not -using the same convention (files are named `YYYYMMDD` corresponding to -the date jobs ended), do not use this option. diff --git a/docs/shredder.md b/docs/shredder.md index 112ce01139..51020cf0ca 100644 --- a/docs/shredder.md +++ b/docs/shredder.md @@ -97,15 +97,8 @@ note that this is **not** currently supported for cloud and storage files: $ xdmod-shredder -i file ... -An entire directory of files can be shredded, but the names of the files -must be formatted as `YYYYMMDD` (e.g. 20120101). - -The log file for the current day will be ignored (along with any files -that correspond to future dates). This is intended to prevent the -shredding of partial log files. - -If the database is empty all files that meet the above constraints will -be shredded. If there is data in the database, only files dated after -the date of the most recent job will be shredded. +An entire directory of files may be shredded. For all job formats other than +`pbs` (see [PBS Notes](resource-manager-pbs.html) for details) this will shred +every file in the directory. Cloud and storage files must end in `.json`. $ xdmod-shredder -d directory ...