From c2e4efac2c2f44f9f230d0667959703378e9dc2c Mon Sep 17 00:00:00 2001 From: Jeanette Sperhac Date: Thu, 11 Mar 2021 09:26:14 -0500 Subject: [PATCH 1/7] Create gateways realm installation instructions --- docs/gateways-realm.md | 138 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 docs/gateways-realm.md diff --git a/docs/gateways-realm.md b/docs/gateways-realm.md new file mode 100644 index 0000000000..fabd13ca39 --- /dev/null +++ b/docs/gateways-realm.md @@ -0,0 +1,138 @@ +# Enabling the Gateways realm +Open XDMoD 9.5 + +The gateways realm displays jobs information due to science gateways. It does this by incorporating jobs data in a new XDMoD database, modw_gateways, created for capturing gateways data. This realm is not enabled by default in the current release. These instructions show you how to enable and populate the gateways realm in your Open XDMoD installation. + +## 1. Database configuration + +The gateways realm displays jobs information on science gateways for an Open XDMoD installation. It incorporates jobs that already exist in your Open XDMoD database. To do so, you will indicate the desired gateways jobs in the ingestion step indicated in b. below. + +### etl_overseer.php + +The database configuration steps are accomplished by running the script `etl_overseer.php` with different options. You should specify this script's full path when calling it (likely `/usr/share/xdmod/tools/etl/`, but check your system). This script accepts numerous parameters described in the help: + +`etl_overseer.php --help` + +### a. Bootstrap the gateways realm + +This first step of Gateways database configuration creates the modw_gateways database, and some needed tables. + +Command: + +`etl_overseer.php -p gateways.bootstrap` + +Verify: + +Successful execution results in creation of the modw_gateways database schema with empty enduser, gateway, gatewayfact_by_day_joblist, and job_metadata tables. + +### b. Select and ingest jobs data for the modw_gateways database + +The ingestion step supplies the name of the community user (gateway proxy user) that gateways use to run jobs on your resource, and joins it to the `modw.person` table. This means that the jobs submitted by that community user will be aggregated for the Gateways realm. You can change this query to match your needs, such as specifying multiple community usernames, or restricting a different column from the person table. + +To change the specification of gateways jobs to aggregate, edit the file `etl/etl_action_defs.d/gateways/gateway.json`, and supply a different where clause. The current where clause in the file is: + +``` + "where": [ + "p.last_name = '${community-user}'" +``` + +In its current state, the WHERE condition accepts as a parameter the gateway community user's surname. The command below specifies community user having last name 'Bunting'; replace this with an appropriate surname from your `modw.person` table. + +Command: + +`etl_overseer.php -p gateways.ingest -d community-user=Bunting` + +Verify: + +Successful execution results in population of `modw_gateways.gateway` table with records selected from `modw.person` according to the where clause. + +### c. Aggregate the data + +This step creates and populates the joblist, day, month, quarter, and year tables for the gateways realm so that queries over these time periods run quickly. + +Command: + +`etl_overseer.php -m '2001-01-01' -y '2022-01-01' -p gateways.aggregate` + +where: +* -p is "process-section" +* -m is "last-modified-start-date" +* -y is "last-modified-end-date" + +The last-modified-end-date should be interpreted as the date the target records were last changed; it should be greater or equal than the current date. For full aggregation, use 2001-01-01 for -m, or some other date that precedes all jobs in your database. + +Verify: + +Successful execution results in creation and population of the `modw_gateways.gatewayfact*` tables containing aggregated job fact and joblist data. + +Running these steps populates the modw_gateways database needed for the Gateways realm. + +### d. Check the modw_gateways schema + +Querying the Open XDMoD database will now show the modw_gateways schema, containing the following tables and data: + +Table Name | Description +-----------|------------- + enduser | empty, reserved for future development + gateway | one record per gateway (from modw.Person) + gatewayfact_by_day | jobs aggregated by day + gatewayfact_by_day_joblist | joblist by day + gatewayfact_by_month | jobs aggregated by month + gatewayfact_by_quarter | jobs aggregated by quarter + gatewayfact_by_year| jobs aggregated by year + job_metadata | empty, reserved for future development + + +## 2. Edit resource_types.json + +The resource configuration file at `/etc/xdmod/resource_types.json` must be edited to include the Gateways realm (data sources). This change configures the UI to display the Gateways realm and its metrics. + +You may add the Gateways realm anyplace the Jobs realm is specified in the file. At a minimum, add the string "Gateways" to the HPC object, as follows: + +``` +{ + "resource_types": { + "HPC": { + "description": "High-performance computing", + "realms": [ + "Jobs", + "Gateways" + ] + }, +... +} +``` +### 3. Run acl-config + +This step enables the UI to display the new Gateways realm that you configured in `resource_types.json`. + +Command: + +`acl-config` + +### 4. Restart services + +For good measure you may now restart all services (database, webserver, etc.). + +### 5. Verify + +To verify the new realm is present, refresh XDMoD in the browser. You should now see: + +- Usage tab includes Gateways Summary metrics in the Metrics listing. +- These Gateways usage metrics with your data can be selected, plotted, and drilled down + +- Gateways metrics are available for plotting, drilldown, etc. in Metric Explorer +- Following two drilldowns, Show Raw Data is available for Gateways data, + enabling selection and review of job accounting data in the Job Viewer tab. + +- Gateways data are available for export in the Data Export tab. + +## Future work + +Further work is planned to enhance the Gateways realm. It includes: + +- automate this process of installing the gateways realm +- incorporate additional job metadata into the gateways schema +- incorporate gateway enduser metadata into the gateways realm +- enable search and drill-down by gateway enduser and other gateways metadata + From a21fa444310c99c7cc6991beec400d2652c7685e Mon Sep 17 00:00:00 2001 From: Jeanette Sperhac Date: Thu, 11 Mar 2021 09:59:16 -0500 Subject: [PATCH 2/7] gateways-realm.md --- docs/gateways-realm.md | 69 ++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 36 deletions(-) diff --git a/docs/gateways-realm.md b/docs/gateways-realm.md index fabd13ca39..aa3259c3fb 100644 --- a/docs/gateways-realm.md +++ b/docs/gateways-realm.md @@ -12,20 +12,20 @@ The gateways realm displays jobs information on science gateways for an Open XDM The database configuration steps are accomplished by running the script `etl_overseer.php` with different options. You should specify this script's full path when calling it (likely `/usr/share/xdmod/tools/etl/`, but check your system). This script accepts numerous parameters described in the help: `etl_overseer.php --help` - -### a. Bootstrap the gateways realm + +### a. Bootstrap the gateways realm This first step of Gateways database configuration creates the modw_gateways database, and some needed tables. - + Command: - + `etl_overseer.php -p gateways.bootstrap` - + Verify: Successful execution results in creation of the modw_gateways database schema with empty enduser, gateway, gatewayfact_by_day_joblist, and job_metadata tables. - -### b. Select and ingest jobs data for the modw_gateways database + +### b. Select and ingest jobs data for the modw_gateways database The ingestion step supplies the name of the community user (gateway proxy user) that gateways use to run jobs on your resource, and joins it to the `modw.person` table. This means that the jobs submitted by that community user will be aggregated for the Gateways realm. You can change this query to match your needs, such as specifying multiple community usernames, or restricting a different column from the person table. @@ -39,29 +39,29 @@ To change the specification of gateways jobs to aggregate, edit the file `etl/et In its current state, the WHERE condition accepts as a parameter the gateway community user's surname. The command below specifies community user having last name 'Bunting'; replace this with an appropriate surname from your `modw.person` table. Command: - + `etl_overseer.php -p gateways.ingest -d community-user=Bunting` -Verify: +Verify: Successful execution results in population of `modw_gateways.gateway` table with records selected from `modw.person` according to the where clause. - + ### c. Aggregate the data -This step creates and populates the joblist, day, month, quarter, and year tables for the gateways realm so that queries over these time periods run quickly. +This step creates and populates the joblist, day, month, quarter, and year tables for the gateways realm so that queries over these time periods run quickly. Command: - + `etl_overseer.php -m '2001-01-01' -y '2022-01-01' -p gateways.aggregate` -where: +where: * -p is "process-section" * -m is "last-modified-start-date" -* -y is "last-modified-end-date" - +* -y is "last-modified-end-date" + The last-modified-end-date should be interpreted as the date the target records were last changed; it should be greater or equal than the current date. For full aggregation, use 2001-01-01 for -m, or some other date that precedes all jobs in your database. -Verify: +Verify: Successful execution results in creation and population of the `modw_gateways.gatewayfact*` tables containing aggregated job fact and joblist data. @@ -73,14 +73,14 @@ Querying the Open XDMoD database will now show the modw_gateways schema, contain Table Name | Description -----------|------------- - enduser | empty, reserved for future development - gateway | one record per gateway (from modw.Person) - gatewayfact_by_day | jobs aggregated by day - gatewayfact_by_day_joblist | joblist by day - gatewayfact_by_month | jobs aggregated by month - gatewayfact_by_quarter | jobs aggregated by quarter - gatewayfact_by_year| jobs aggregated by year - job_metadata | empty, reserved for future development +enduser | empty, reserved for future development +gateway | one record per gateway (from modw.Person) +gatewayfact_by_day | jobs aggregated by day +gatewayfact_by_day_joblist | joblist by day +gatewayfact_by_month | jobs aggregated by month +gatewayfact_by_quarter | jobs aggregated by quarter +gatewayfact_by_year| jobs aggregated by year +job_metadata | empty, reserved for future development ## 2. Edit resource_types.json @@ -102,17 +102,17 @@ You may add the Gateways realm anyplace the Jobs realm is specified in the file. ... } ``` -### 3. Run acl-config +### 3. Run acl-config This step enables the UI to display the new Gateways realm that you configured in `resource_types.json`. Command: `acl-config` - -### 4. Restart services -For good measure you may now restart all services (database, webserver, etc.). +### 4. Restart services + +For good measure you may now restart all services (database, webserver, etc.). ### 5. Verify @@ -120,19 +120,16 @@ To verify the new realm is present, refresh XDMoD in the browser. You should now - Usage tab includes Gateways Summary metrics in the Metrics listing. - These Gateways usage metrics with your data can be selected, plotted, and drilled down - - Gateways metrics are available for plotting, drilldown, etc. in Metric Explorer -- Following two drilldowns, Show Raw Data is available for Gateways data, - enabling selection and review of job accounting data in the Job Viewer tab. - -- Gateways data are available for export in the Data Export tab. - +- Following two drilldowns, Show Raw Data is available for Gateways data, enabling selection and review of job accounting data in the Job Viewer tab. +- Gateways data are available for export in the Data Export tab. + ## Future work Further work is planned to enhance the Gateways realm. It includes: - automate this process of installing the gateways realm - incorporate additional job metadata into the gateways schema -- incorporate gateway enduser metadata into the gateways realm +- incorporate gateway enduser metadata into the gateways realm - enable search and drill-down by gateway enduser and other gateways metadata - + From 971c70d3684ce929aa9144773f933707d6564563 Mon Sep 17 00:00:00 2001 From: Jeanette Sperhac Date: Fri, 12 Mar 2021 11:52:27 -0500 Subject: [PATCH 3/7] Whitespace --- docs/gateways-realm.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/gateways-realm.md b/docs/gateways-realm.md index aa3259c3fb..5d4dc71ee2 100644 --- a/docs/gateways-realm.md +++ b/docs/gateways-realm.md @@ -132,4 +132,3 @@ Further work is planned to enhance the Gateways realm. It includes: - incorporate additional job metadata into the gateways schema - incorporate gateway enduser metadata into the gateways realm - enable search and drill-down by gateway enduser and other gateways metadata - From b5fa987f757b3cf0b9c009df063c31cb9a0d6c5e Mon Sep 17 00:00:00 2001 From: Jeanette Sperhac Date: Fri, 12 Mar 2021 11:55:37 -0500 Subject: [PATCH 4/7] whitespace --- docs/gateways-realm.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/gateways-realm.md b/docs/gateways-realm.md index aa3259c3fb..5d4dc71ee2 100644 --- a/docs/gateways-realm.md +++ b/docs/gateways-realm.md @@ -132,4 +132,3 @@ Further work is planned to enhance the Gateways realm. It includes: - incorporate additional job metadata into the gateways schema - incorporate gateway enduser metadata into the gateways realm - enable search and drill-down by gateway enduser and other gateways metadata - From b7e0d7999760177025292b55ef472dac646041cf Mon Sep 17 00:00:00 2001 From: Jeanette Sperhac Date: Fri, 12 Mar 2021 14:52:39 -0500 Subject: [PATCH 5/7] Describe ID of gateway HPC jobs Co-authored-by: Joe White --- docs/gateways-realm.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/gateways-realm.md b/docs/gateways-realm.md index 5d4dc71ee2..8bc837f8e4 100644 --- a/docs/gateways-realm.md +++ b/docs/gateways-realm.md @@ -26,6 +26,10 @@ Verify: Successful execution results in creation of the modw_gateways database schema with empty enduser, gateway, gatewayfact_by_day_joblist, and job_metadata tables. ### b. Select and ingest jobs data for the modw_gateways database +XDMoD needs a mechanism to identify HPC jobs that were run via a gateway. The default mechanism is to use the last name associated with the user account that ran the jobs. Instructions for configuring the names are in the (User/PI Names guide)[user-names.md]. The steps to use this mechanism are: +1) Choose a suitable identifier to be used for a gateway (such as 'Gateway Proxy User') +1) Update the last name entries in `names.csv` for each gateway proxy user account. +1) Import the updated `names.csv` The ingestion step supplies the name of the community user (gateway proxy user) that gateways use to run jobs on your resource, and joins it to the `modw.person` table. This means that the jobs submitted by that community user will be aggregated for the Gateways realm. You can change this query to match your needs, such as specifying multiple community usernames, or restricting a different column from the person table. From 5215d02dab3e45d3a20cc34e0d0908f17aadae3d Mon Sep 17 00:00:00 2001 From: Jeanette Sperhac Date: Fri, 12 Mar 2021 15:12:57 -0500 Subject: [PATCH 6/7] Requested updates to Gateways docs --- docs/_data/toc.yml | 2 ++ docs/gateways-realm.md | 16 +++++++++------- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/_data/toc.yml b/docs/_data/toc.yml index 8143febb8c..933d3e1603 100644 --- a/docs/_data/toc.yml +++ b/docs/_data/toc.yml @@ -43,6 +43,8 @@ toc: url: /simpleSAMLphp-ldap.html - page: Upgrade Guide url: /upgrade.html + - page: Gateways Realm (alpha) + url: /gateways-realm.html - title: Using subfolderitems: - page: Shredder Guide diff --git a/docs/gateways-realm.md b/docs/gateways-realm.md index 5d4dc71ee2..ce07a9e318 100644 --- a/docs/gateways-realm.md +++ b/docs/gateways-realm.md @@ -1,5 +1,4 @@ # Enabling the Gateways realm -Open XDMoD 9.5 The gateways realm displays jobs information due to science gateways. It does this by incorporating jobs data in a new XDMoD database, modw_gateways, created for capturing gateways data. This realm is not enabled by default in the current release. These instructions show you how to enable and populate the gateways realm in your Open XDMoD installation. @@ -9,9 +8,12 @@ The gateways realm displays jobs information on science gateways for an Open XDM ### etl_overseer.php -The database configuration steps are accomplished by running the script `etl_overseer.php` with different options. You should specify this script's full path when calling it (likely `/usr/share/xdmod/tools/etl/`, but check your system). This script accepts numerous parameters described in the help: +The database configuration steps are accomplished by running the script `etl_overseer.php` with +different options. You should specify this script's full path when calling it (RPM install location +is `/usr/share/xdmod/tools/etl/`). This script accepts numerous parameters described in the help +output: -`etl_overseer.php --help` +`$ /usr/share/xdmod/tools/etl/etl_overseer.php --help` ### a. Bootstrap the gateways realm @@ -19,7 +21,7 @@ This first step of Gateways database configuration creates the modw_gateways dat Command: -`etl_overseer.php -p gateways.bootstrap` +`$ /usr/share/xdmod/tools/etl/etl_overseer.php -p gateways.bootstrap` Verify: @@ -40,7 +42,7 @@ In its current state, the WHERE condition accepts as a parameter the gateway com Command: -`etl_overseer.php -p gateways.ingest -d community-user=Bunting` +`$ /usr/share/xdmod/tools/etl/etl_overseer.php -p gateways.ingest -d community-user=Bunting` Verify: @@ -52,7 +54,7 @@ This step creates and populates the joblist, day, month, quarter, and year table Command: -`etl_overseer.php -m '2001-01-01' -y '2022-01-01' -p gateways.aggregate` +`$ /usr/share/xdmod/tools/etl/etl_overseer.php -m '2001-01-01' -y '2022-01-01' -p gateways.aggregate` where: * -p is "process-section" @@ -108,7 +110,7 @@ This step enables the UI to display the new Gateways realm that you configured i Command: -`acl-config` +`$ acl-config` ### 4. Restart services From 11e3706a3f2e3f1fde4777abd6133b62372c0698 Mon Sep 17 00:00:00 2001 From: Jeanette Sperhac Date: Fri, 19 Mar 2021 11:19:14 -0400 Subject: [PATCH 7/7] Small changes addressing review feedback --- docs/gateways-realm.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/gateways-realm.md b/docs/gateways-realm.md index 5506b30baa..1153721a43 100644 --- a/docs/gateways-realm.md +++ b/docs/gateways-realm.md @@ -1,6 +1,6 @@ # Enabling the Gateways realm -The gateways realm displays jobs information due to science gateways. It does this by incorporating jobs data in a new XDMoD database, modw_gateways, created for capturing gateways data. This realm is not enabled by default in the current release. These instructions show you how to enable and populate the gateways realm in your Open XDMoD installation. +The gateways realm displays jobs information due to science gateways. It does this by incorporating jobs data in a new XDMoD database, `modw_gateways`, created for capturing gateways data. This realm is not enabled by default in the current release. These instructions show you how to enable and populate the gateways realm in your Open XDMoD installation. ## 1. Database configuration @@ -42,11 +42,11 @@ To change the specification of gateways jobs to aggregate, edit the file `etl/et "p.last_name = '${community-user}'" ``` -In its current state, the WHERE condition accepts as a parameter the gateway community user's surname. The command below specifies community user having last name 'Bunting'; replace this with an appropriate surname from your `modw.person` table. +In its current state, the WHERE condition accepts as a parameter the gateway community user's `last_name`. The command below specifies community user 'Gateway proxy user'; replace this with an appropriate `last_name` from your `modw.person` table. Command: -`$ /usr/share/xdmod/tools/etl/etl_overseer.php -p gateways.ingest -d community-user=Bunting` +`$ /usr/share/xdmod/tools/etl/etl_overseer.php -p gateways.ingest -d community-user='Gateway proxy user'` Verify: @@ -116,11 +116,7 @@ Command: `$ acl-config` -### 4. Restart services - -For good measure you may now restart all services (database, webserver, etc.). - -### 5. Verify +### 4. Verify To verify the new realm is present, refresh XDMoD in the browser. You should now see: