Add cloud instance viewer #1473

eiffel777 · 2020-12-23T19:42:35Z

This PR adds functionality to track both active and inactive sessions and adds to the Job Viewer the ability to view details about a particular instance including all of the sessions of a VM on a timeline showing when they are active and inactive. Data for the Cloud realm should be re-ingested since we are tracking some new events.

The tests have changed to include the cloud realm in any test that checks to see what raw data realms exist. The regression tests have changed because stop events for an active session are no longer the .end event such as compute.instance.power_off.end but are now the .start events like compute.instance.power_off.start. The time difference between these events in the test data is a few seconds slightly changing the core hours and wall hours.

Motivation and Context

It is useful to see the lifecycle of a VM and also let's us track data about inactive sessions in the future.

Tests performed

Tested locally and on metrics-dev.

Checklist:

The pull request description is suitable for a Changelog entry
The milestone is set correctly on the pull request
The appropriate labels have been added to the pull request

…oud-instance-viewer

…nstances

… for use with instance viewer

jpwhite4 · 2021-01-05T23:04:45Z

classes/ETL/Ingestor/CloudStateReconstructorTransformIngestor.php

-        } elseif (in_array($srcRecord['event_type_id'], $this->_start_event_ids)) {
-            $this->updateInstance($srcRecord);
-        } elseif (in_array($srcRecord['event_type_id'], $this->_stop_event_ids)) {
+        } elseif (in_array($srcRecord['event_type_id'], [1,16])) {


The hardcoded 1,16 seems out of place here since you have gone to the trouble of making the event ids php constants and then putting them in class member variables.

jpwhite4 · 2021-01-06T01:04:47Z

classes/ETL/Ingestor/CloudStateReconstructorTransformIngestor.php

+          self::SHELVE,
+          self::UNSHELVE,
+          self::POWER_OFF_START,
+          self::POWER_OFF,


I think this code is going to need some explanatory comments. How is POWER_OFF a start event? How come the start_event_ids and end_event_ids are exactly the same except for TERMINATE is not a start event?

jpwhite4 · 2021-01-06T01:05:39Z

classes/ETL/Ingestor/CloudStateReconstructorTransformIngestor.php

        $this->_end_time = $etlConfig->getVariableStore()->endDate ? date('Y-m-d H:i:s', strtotime($etlConfig->getVariableStore()->endDate)) : null;

        $this->resetInstance();
    }

    private function initInstance($srcRecord)
    {
-        $default_end_time = isset($this->_end_time) ? $this->_end_time : $srcRecord['event_time_ts'];
+        $beginOfDay = strtotime("today", $srcRecord['event_time_ts']);


Is this 'timezone' safe?

…oud-instance-viewer

jsperhac · 2021-01-15T17:14:45Z

tests/artifacts/xdmod/component/export/realm_manager/output/realms-for-user.json

-        [
-            {
+        {
+            "0": {


Note that recently merged PR #1481 re-indexes the arrays returned from getRealms() so that explicit indices are not needed here (previously there were potentially gaps in the array keys returned). This also applies to the realms.json testing artifact.

ryanrath

Alrighty, here's my first go through. Ultimately it looks good, most of things I found were with an eye to make our future selves hate our past selves a little less. The only thing of any substance that I found / thought of is the whole having to ultimately support multiple Cloud infrastructures thing that I think is more of a point of discussion and definitely not a show stopper.

ryanrath · 2021-02-24T15:32:19Z

classes/ETL/Ingestor/CloudStateReconstructorTransformIngestor.php

+    const UNPAUSE_START = 56;
+    const POWER_ON_START = 58;
+    const UNSUSPEND_START = 60;
+    const UNSHELVE_END = 63;
    const START = 2;
    const RESUME = 8;
    const STATE_REPORT = 16;
    const UNSHELVE = 20;
    const UNPAUSE = 57;


Is this event type used? I noticed that this variable wasn't referenced by anything.

ryanrath · 2021-02-24T15:48:24Z

configuration/etl/etl_action_defs.d/cloud_common/cloud_metrics_aggregation.json

              }
            ],
            "where": [
              "sr.start_day_id <= ${:PERIOD_END_DAY_ID} AND sr.end_day_id >= ${:PERIOD_START_DAY_ID}",
              "sr.instance_type != \"Unknown\"",
              "sr.instance_id !=\"1\"",
-              "sr.num_cores != 0"
+              "sr.num_cores != 0",
+              "sr.start_event_type_id IN (2,8,16,20,57,59,61)"


Could we get a comment w/ an explanation of where these numbers are pulled from, or where their definitions can be found? I'm guessing it's the CloudStateReconstructorTransformIngestor? But I'd imagine that'll make it easier for somebody to know how to go further down the rabbit hole in the future.

ryanrath · 2021-02-24T15:52:25Z

configuration/etl/etl_action_defs.d/cloud_common/cloud_metrics_aggregation_by_day.json

+            ],
+            "where": [
+              "sr.start_day_id <= ${:PERIOD_END_DAY_ID} AND sr.end_day_id >= ${:PERIOD_START_DAY_ID}",
+              "sr.instance_type != \"Unknown\"",


It's not really a big deal, but it might be more readable if these were single quotes so as not to have to deal with escaping. Same for the line below this one.

ryanrath · 2021-02-24T15:54:34Z

configuration/etl/etl_action_defs.d/cloud_openstack/root_volume.json

@@ -14,7 +14,7 @@
    "source_query": {
        "records": {
            "resource_id": "staging.resource_id",
-            "asset_type_id": "staging.root_volume_type_id",
+            "asset_type_id": 2,


In the same vein as one of my comments above, a quick comment about what asset_type_id: 2 is in a human readable format would go a long way towards future readability.

ryanrath · 2021-02-24T15:54:48Z

configuration/etl/etl_action_defs.d/cloud_openstack/volume.json

@@ -11,6 +11,7 @@
        "records": {
            "resource_id": "raw.resource_id",
            "account_id": "act.account_id",
+            "asset_type_id": 1,


See above comment about future readability.

ryanrath · 2021-02-24T16:06:35Z

html/gui/js/modules/job_viewer/VMStateChartPanel.js

+        },
+        load_record: function (panel, record) {
+            var self = this;
+            var active_states = [2, 8, 16, 20, 57, 59, 61];


So I'm wondering if it might be worth it to have a this data reside within a config file so that the data could be shared between the front & back end without having to manually make code changes? This obviously works but just a thought to maybe make any future changes easier. Another thing to keep in mind is that we're going to end up having to support multiple cloud's and each one is going to have different states etc. Maybe that's not something we worry about now, but it's something that we will have to address.

eiffel777 added 16 commits December 1, 2020 15:27

adding support to view the lifecycle of a VM

bf7fcf6

adding plotly to composer.lock file

c442708

Merge branch 'xdmod9.5' of https://github.com/ubccr/xdmod into add-cl…

21bcd64

…oud-instance-viewer

changing how the plotly legend is displayed

d1abfe1

plotly was setting the wrong date range for the date slider in some i…

39ba61a

…nstances

adding lines to denote where a session starts

747180e

renaming cloud raw statistics file

218c32a

changing 1 week range slider button to be 7 days

688158f

fixing merge issues

78f65a4

adding changes to tests and adding cloudfact_by_day_sessionlist table…

f119f93

… for use with instance viewer

updating tests to account for new raw data realm

be12216

code cleanup

5258f36

documentation additions and adding migration files

13809ef

cleanup before PR

cc5c387

making sure composer.lock only adds the plotly library

a8fa49d

event and migration updates

edcc4f2

eiffel777 added Category:Cloud Cloud Realm new feature New functionality labels Dec 23, 2020

eiffel777 added this to the 9.5.0 milestone Dec 23, 2020

eiffel777 requested review from jpwhite4 and ryanrath December 23, 2020 19:42

eiffel777 self-assigned this Dec 23, 2020

Merge branch 'xdmod9.5' into add-cloud-instance-viewer

c476a0c

eiffel777 requested a review from jtpalmer December 23, 2020 19:43

eiffel777 added 6 commits December 26, 2020 11:20

style and chartFilterTest fixes

50c72de

style fixes

83a4c98

more style fixes

b9355e0

more style fixes

0d7dd2b

style fixes

2c45d22

style changes

a928c90

eiffel777 added 3 commits December 26, 2020 13:20

php style tests

e65fd28

php style fixes

e2715ca

fixing issues with upgrade

d9a8435

jpwhite4 reviewed Jan 5, 2021

View reviewed changes

jpwhite4 reviewed Jan 6, 2021

View reviewed changes

eiffel777 added 4 commits January 7, 2021 13:47

Merge branch 'xdmod9.5' of https://github.com/ubccr/xdmod into add-cl…

c5ff6ef

…oud-instance-viewer

updates to CloudStateReconstructorTransformer

cb7ebb1

Merge branch 'xdmod9.5' of https://github.com/ubccr/xdmod into add-cl…

fbfcd6b

…oud-instance-viewer

updating component tests

9b867fd

jsperhac reviewed Jan 15, 2021

View reviewed changes

jsperhac mentioned this pull request Jan 15, 2021

Populate joblist table for gateway jobs #1411

Merged

6 tasks

eiffel777 added 2 commits February 22, 2021 11:29

merging recent 9.5 update in

6bd55c8

updating to use new logger class

c6d903b

ryanrath reviewed Feb 24, 2021

View reviewed changes

eiffel777 added 5 commits February 25, 2021 14:20

merging in latest 9.5 changes

b5a67ff

fixing tests

8bfede2

updating comments for clarity

0e3f1b5

merging updates from latest xdmod9.5

f4f9d21

fix style issues

415cd15

ryanrath self-requested a review March 1, 2021 18:29

ryanrath approved these changes Mar 1, 2021

View reviewed changes

Merge branch 'xdmod9.5' into add-cloud-instance-viewer

53aa173

eiffel777 merged commit 4b27cb1 into ubccr:xdmod9.5 Mar 2, 2021

jtpalmer mentioned this pull request Jun 18, 2021

Add plotly to .gitignore #1538

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cloud instance viewer #1473

Add cloud instance viewer #1473

eiffel777 commented Dec 23, 2020

jpwhite4 Jan 5, 2021

jpwhite4 Jan 6, 2021

jpwhite4 Jan 6, 2021

jsperhac Jan 15, 2021

ryanrath left a comment

ryanrath Feb 24, 2021

ryanrath Feb 24, 2021

ryanrath Feb 24, 2021

ryanrath Feb 24, 2021

ryanrath Feb 24, 2021

ryanrath Feb 24, 2021

Add cloud instance viewer #1473

Add cloud instance viewer #1473

Conversation

eiffel777 commented Dec 23, 2020

Motivation and Context

Tests performed

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanrath left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment