IGNITE-13860 Provide the tool to build cluster performance report #16

NSAmelchev · 2020-06-10T12:17:33Z

Cluster profiling tool based on logging performance statistics to files. See apache/ignite#7693

Run command to build the tool: mvn clean install
Release archive will be be under the directory: modules/performance-statistics-ext/target/release-package-ignite-performance-statistics.zip

To collect statistics in runtime and to build the performance report follow:

Start collecting performance statistics in the cluster. See PerformanceStatisticsMBean
Collect workload statistics.
Stopcollecting performance statistics in the cluster.. See PerformanceStatisticsMBean
Collect performance statistics files from all nodes under an empty directory.
Run script build-report.sh path_to_files to build the performance report.

The report contains statistics:

cache operations: get|getAll|put|putAll|remove|removeAll|getAndPut|getAndRemove|lock|invoke|invokeAll
transaction commit, rollback; histogram of transaction durations
SQL, SCAN queries, top of slowest queries
Tasks and jobs. Top of slowest tasks
cluster info

modules/profiling-ext/README.txt

samaitra

I observed we have an open PR in ignite repo for this feature with different set of changes.

apache/ignite#7693

Can you please share more info on how we can use the profiling tool with ignite-extensions modules?

samaitra · 2020-06-21T19:11:54Z

modules/profiling-ext/README.txt

@@ -0,0 +1,48 @@
+Apache Ignite Profiling Module


Please add license header

The project's readme-files does not contain a license header. Why we should add it?

samaitra · 2020-06-21T19:12:56Z

modules/profiling-ext/assembly/dependencies-profiling.xml

+          http://maven.apache.org/xsd/assembly-1.1.2.xsd">
+    <id>profiling-report-sources</id>
+
+    <includeBaseDirectory>false</includeBaseDirectory>


can you please share details on this base dir flag

I have excluded base directory from UI sources archive:
report.zip/{sources}
instead of
report.zip/report/{sources}

samaitra · 2020-06-21T20:19:59Z

modules/profiling-ext/report/libs/Chart.min.js

+ * Version: 2.7.3
+ *
+ * Copyright 2018 Chart.js Contributors
+ * Released under the MIT license


Are we ok to use mix of MIT and Apache license?

It's ok, I have mentioned these components in the LICENSE file.

samaitra · 2020-06-21T20:23:36Z

modules/profiling-ext/report/libs/jquery-3.3.1.min.js

@@ -0,0 +1,2 @@
+/*! jQuery v3.3.1 | (c) JS Foundation and other contributors | jquery.org/license */


We probably need to add license header in this file

I have clarified the license.

samaitra · 2020-06-21T20:30:04Z

.../profiling-ext/src/main/java/org/apache/ignite/internal/profiling/handlers/QueryHandler.java

+        Query query = new Query(type, text, queryNodeId, id, startTime, duration, success);
+
+        OrderedFixedSizeStructure<Long, Query> tree = topSlow.computeIfAbsent(type,
+            t -> new OrderedFixedSizeStructure<>());


can we rename t to a variable name

samaitra · 2020-06-21T20:30:19Z

.../profiling-ext/src/main/java/org/apache/ignite/internal/profiling/handlers/QueryHandler.java

+        long physicalReads) {
+
+        Map<Long, long[]> ids = readsById.computeIfAbsent(type, t -> new HashMap<>())
+            .computeIfAbsent(queryNodeId, k -> new HashMap<>());


can we rename k to a variable

samaitra

reviewed and shared comments

samaitra

Looks good

samaitra · 2020-07-12T21:16:45Z

@NSAmelchev There is conflicts in this PR, can you please resolve the conflict and then go ahead and merge the changes.

# Conflicts: # pom.xml

modules/performance-statistics-ext/pom.xml

nizhikov · 2020-12-22T09:29:34Z

modules/performance-statistics-ext/bin/build-report.sh

+#
+# Run tool.
+#
+Java ${JVM_OPTS} -cp "${CP}" ${MAIN_CLASS} $@


typo: Java -> java

nizhikov · 2020-12-22T09:40:03Z

modules/performance-statistics-ext/pom.xml

+                                <resource>
+                                    <directory>${basedir}/target</directory>
+                                    <includes>
+                                        <include>report.zip</include>


Can we include UI resources as a directory not zip archive?

nizhikov · 2020-12-22T10:31:28Z

@NSAmelchev Can we modify build process so it will download css, js resources in the build time?
I think we shouldn't store css or js resources in Ignite repository.

NSAmelchev added 2 commits June 10, 2020 15:15

Ignite profiling. WIP

b6bd498

WIP

9acc508

samaitra reviewed Jun 11, 2020

View reviewed changes

modules/profiling-ext/README.txt Outdated Show resolved Hide resolved

samaitra reviewed Jun 11, 2020

View reviewed changes

WIP. Fix maven build. Add test.

555a4c9

NSAmelchev marked this pull request as draft June 11, 2020 08:04

WIP. Remove CDN links. Offline package

8d9b78b

NSAmelchev mentioned this pull request Jun 15, 2020

IGNITE-12666 Provide cluster performance profiling tool apache/ignite#7693

Merged

Fix typo and codestyle

f481ef0

NSAmelchev marked this pull request as ready for review June 15, 2020 07:52

samaitra reviewed Jun 21, 2020

View reviewed changes

samaitra suggested changes Jun 21, 2020

View reviewed changes

NSAmelchev added 3 commits June 22, 2020 12:42

Review fixes

2b8a554

Review fixes

b60523d

Review fixes (License)

f25001c

samaitra approved these changes Jun 24, 2020

View reviewed changes

NSAmelchev marked this pull request as draft July 13, 2020 06:34

NSAmelchev added 7 commits August 13, 2020 11:58

Core module fixes 1

d4d1c4e

WIP

0bc20d9

WIP

41f03bc

WIP

1fad9c5

Merge branch 'master' into ignite-12666

01bd2a6

# Conflicts: # pom.xml

WIP

3bba76d

WIP

514c686

Minor fixes

d8701fc

NSAmelchev marked this pull request as ready for review December 3, 2020 08:54

nizhikov reviewed Dec 14, 2020

View reviewed changes

modules/performance-statistics-ext/pom.xml Show resolved Hide resolved

NSAmelchev added 2 commits December 15, 2020 09:56

Fix scope

27e8e1b

Add suite

b3d1efb

NSAmelchev changed the title ~~IGNITE-12666 Provide cluster performance profiling tool~~ IGNITE-13860 Provide the tool to build cluster performance report Dec 16, 2020

NSAmelchev added 3 commits December 16, 2020 10:02

Fix test

0bf872f

Fix select

bc9d38e

Merge branch 'master' into ignite-12666

c8936cf

nizhikov reviewed Dec 22, 2020

View reviewed changes

nizhikov and others added 3 commits December 22, 2020 13:39

Update CacheOperationsHandler.java

83f8886

Update TransactionsHandler.java

7cae7c7

Fix typo

34a9b71

nizhikov merged commit fd7aafd into apache:master Dec 22, 2020

NSAmelchev deleted the ignite-12666 branch April 7, 2021 12:07

		@@ -0,0 +1,2 @@
		/! jQuery v3.3.1 \| (c) JS Foundation and other contributors \| jquery.org/license /

IGNITE-13860 Provide the tool to build cluster performance report #16

IGNITE-13860 Provide the tool to build cluster performance report #16

Uh oh!

Conversation

NSAmelchev commented Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

samaitra left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samaitra left a comment

Choose a reason for hiding this comment

Uh oh!

samaitra left a comment

Choose a reason for hiding this comment

Uh oh!

samaitra commented Jul 12, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nizhikov commented Dec 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

NSAmelchev commented Jun 10, 2020 •

edited

Loading