Update shredder documentation

bin/xdmod-shredder

@@ -296,11 +296,15 @@ Usage: xdmod-shredder [-v] -r resource -f format [-i file|-d dir]
         storage data.
 
     -d, --dir *directory*
-        Specify a directory containing log files to shred.  Log files in
-        the directory must be named in YYYYMMDD format where the file
-        name corresponds to the end date of the jobs it contains.  Only
-        files with dates that are more recent than the end date of the
-        most recent job will be shredded.
+        Specify a directory containing log files to shred.
+
+        For PBS format log files, files in the directory must be named in
+        YYYYMMDD format where the file name corresponds to the end date of the
+        jobs it contains.  Only files with dates that are more recent than the
+        end date of the most recent job will be shredded.
+
+        For all other formats, all files in the directory will be shredded.
+        Cloud and storage files must end in ".json".
 
     --host-filter *host-regex*
         Specify the host filter regex for jobs.  Any job that does not

docs/resource-manager-lsf.md

@@ -17,11 +17,3 @@ PI Data Source
 
 Open XDMoD uses the `projectName` field from `lsb.acct` as the default
 PI data source.
-
-Unsupported Shredder Features
------------------------------
-
-The xdmod-shredder `-d`/`--dir` option was designed to work with the
-accounting log naming convention used by PBS/TORQUE. If you are not
-using the same convention (files are named `YYYYMMDD` corresponding to
-the current date), do not use this option.

docs/resource-manager-pbs.md

@@ -7,11 +7,20 @@ Open XDMoD supports PBS including OpenPBS and TORQUE.
 Logs Files
 ----------
 
-The xdmod-shredder `-d`/`--dir` option was designed to work with the
-accounting log naming convention used by PBS/TORQUE. If you are not
-using the same convention (files are named `YYYYMMDD` corresponding to
-the current date), do not use this option.  These log files are
-typically stored in `/var/spool/pbs/server_priv/accounting`,
-`$PBSROOT/server_priv/accounting`,
+When using the `pbs` format, the `xdmod-shredder` `-d`/`--dir` option expects
+log files to use the accounting log naming convention used by PBS/TORQUE. If
+you are not using the same convention (files are named `YYYYMMDD` corresponding
+to the current date), do not use this option.
+
+The log file for the current day will be ignored (along with any files
+that correspond to future dates). This is intended to prevent the
+shredding of partial log files.
+
+If the database is empty all files that meet the above constraints will
+be shredded. If there is data in the database, only files dated after
+the date of the most recent job will be shredded.
+
+These log files are typically stored in
+`/var/spool/pbs/server_priv/accounting`, `$PBSROOT/server_priv/accounting`,
 `/var/spool/torque/server_priv/accounting`, or
 `$TORQUEROOT/server_priv/accounting`.

docs/resource-manager-sge.md

@@ -32,11 +32,3 @@ data, the unique key constraint on the `shredded_job_sge` table will
 prevent the insertion of this data, but will instead update the primary
 key. This will prevent duplicate data from being entered into the
 database, even though the entire log file will still be parsed.
-
-Unsupported Shredder Features
------------------------------
-
-The xdmod-shredder `-d`/`--dir` option was designed to work with the
-accounting log naming convention used by PBS/TORQUE. If you are not
-using the same convention (files are named `YYYYMMDD` corresponding to
-the current date), do not use this option.

docs/resource-manager-slurm.md

@@ -57,11 +57,3 @@ $ xdmod-shredder -r *cluster* -f slurm -i /tmp/slurm.log
 prevent ambiguities caused by clock changes due to daylight savings. The
 shredder will assume input times are in UTC regardless of your system
 time zone.
-
-Unsupported Shredder Features
------------------------------
-
-The `xdmod-shredder` `-d`/`--dir` option was designed to work with the
-accounting log naming convention used by PBS/TORQUE. If you are not
-using the same convention (files are named `YYYYMMDD` corresponding to
-the date jobs ended), do not use this option.

docs/shredder.md

@@ -97,15 +97,8 @@ note that this is **not** currently supported for cloud and storage files:
 
     $ xdmod-shredder -i file ...
 
-An entire directory of files can be shredded, but the names of the files
-must be formatted as `YYYYMMDD` (e.g. 20120101).
-
-The log file for the current day will be ignored (along with any files
-that correspond to future dates). This is intended to prevent the
-shredding of partial log files.
-
-If the database is empty all files that meet the above constraints will
-be shredded. If there is data in the database, only files dated after
-the date of the most recent job will be shredded.
+An entire directory of files may be shredded.  For all job formats other than
+`pbs` (see [PBS Notes](resource-manager-pbs.html) for details) this will shred
+every file in the directory.  Cloud and storage files must end in `.json`.
 
     $ xdmod-shredder -d directory ...