From eae95194773bebd699ead3d8fd3ff224b4a4b984 Mon Sep 17 00:00:00 2001 From: "Jeffrey T. Palmer" Date: Wed, 15 Jul 2020 14:10:47 -0400 Subject: [PATCH 1/2] Update hierarchy documentation --- docs/hierarchy.md | 42 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 5 deletions(-) diff --git a/docs/hierarchy.md b/docs/hierarchy.md index 1b265c8331..5c62bd0807 100644 --- a/docs/hierarchy.md +++ b/docs/hierarchy.md @@ -7,6 +7,10 @@ populated with site specific data. Importing this data is a two step process. First, you must import the hierarchy items and then you must import a mapping from groups (PIs) to hierarchy items. +For job records the source of PI data depends on the resource manager used and +is configurable. Refer to the [`resources.json` configuration +section](configuration.html#resourcesjson) for more details. + Configuring the Hierarchy ------------------------- @@ -69,7 +73,9 @@ The input CSV would look like this: And imported with a command like this: - $ xdmod-import-csv -t hierarchy -i hierarchy.csv +``` +$ xdmod-import-csv -t hierarchy -i hierarchy.csv +``` After importing the hierarchy it is necessary to provide a mapping from your user groups to the hierarchy items. The input format of this @@ -91,12 +97,38 @@ specified: And imported with a command like this: - $ xdmod-import-csv -t group-to-hierarchy -i group-to-hierarchy.csv +``` +$ xdmod-import-csv -t group-to-hierarchy -i group-to-hierarchy.csv +``` + +After importing the hierarchy data you must re-ingest and re-aggregate any job, +storage, or cloud data for the date range you have already shredded. If you +have tens of millions of records you should run the `xdmod-ingestor` command +multiple times with smaller date ranges to prevent database time-outs. -After importing this data you must ingest it for the date range of any -job data you have already shredded. +This example would re-shred and re-aggregate all data for the year 2012: + +``` +$ xdmod-ingestor --start-date 2012-01-01 --end-date 2012-12-31 +``` + +Reset Hierarchy Data +-------------------- + +If you decided to remove the hierarchy data, or if you would like to replace +your current hierarchy with a different one, a manual process is currently +required. + +Run the following queries in your MySQL database: + +``` +mysql> UPDATE mod_hpcdb.hpcdb_requests SET primary_fos_id = 1; +mysql> DELETE FROM mod_hpcdb.hpcdb_fields_of_science WHERE field_of_science_id != 1; +``` - $ xdmod-ingestor --start-date 2012-01-01 --end-date 2012-12-31 +After that you can import a new hierarchy and mapping if desired. Then +re-ingest and re-aggregate as done in the above section to update the data +warehouse. Disabling Hierarchy Dimensions ------------------------------ From 9cc1add950f507d929482bc9a65ab12db1dbc3a7 Mon Sep 17 00:00:00 2001 From: "Jeffrey T. Palmer" Date: Wed, 15 Jul 2020 14:31:19 -0400 Subject: [PATCH 2/2] Empty commit