Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update hierarchy documentation #1376

Merged
merged 2 commits into from
Jul 15, 2020
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 37 additions & 5 deletions docs/hierarchy.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ populated with site specific data. Importing this data is a two step
process. First, you must import the hierarchy items and then you must
import a mapping from groups (PIs) to hierarchy items.

For job records the source of PI data depends on the resource manager used and
is configurable. Refer to the [`resources.json` configuration
section](configuration.html#resourcesjson) for more details.

Configuring the Hierarchy
-------------------------

Expand Down Expand Up @@ -69,7 +73,9 @@ The input CSV would look like this:

And imported with a command like this:

$ xdmod-import-csv -t hierarchy -i hierarchy.csv
```
$ xdmod-import-csv -t hierarchy -i hierarchy.csv
```

After importing the hierarchy it is necessary to provide a mapping from
your user groups to the hierarchy items. The input format of this
Expand All @@ -91,12 +97,38 @@ specified:

And imported with a command like this:

$ xdmod-import-csv -t group-to-hierarchy -i group-to-hierarchy.csv
```
$ xdmod-import-csv -t group-to-hierarchy -i group-to-hierarchy.csv
```

After importing the hierarchy data you must re-ingest and re-aggregate any job,
storage, or cloud data for the date range you have already shredded. If you
have tens of millions of records you should run the `xdmod-ingestor` command
multiple times with smaller date ranges to prevent database time-outs.

After importing this data you must ingest it for the date range of any
job data you have already shredded.
This example would re-shred and re-aggregate all data for the year 2012:

```
$ xdmod-ingestor --start-date 2012-01-01 --end-date 2012-12-31
```

Reset Hierarchy Data
--------------------

If you decided to remove the hierarchy data, or if you would like to replace
your current hierarchy with a different one, a manual process is currently
required.

Run the following queries in your MySQL database:

```
mysql> UPDATE mod_hpcdb.hpcdb_requests SET primary_fos_id = 1;
mysql> DELETE FROM mod_hpcdb.hpcdb_fields_of_science WHERE field_of_science_id != 1;
```

$ xdmod-ingestor --start-date 2012-01-01 --end-date 2012-12-31
After that you can import a new hierarchy and mapping if desired. Then
re-ingest and re-aggregate as done in the above section to update the data
warehouse.

Disabling Hierarchy Dimensions
------------------------------
Expand Down