Oheger bosch/analyzer/dependency representation

[oheger-bosch]

This PR addresses #3321 by proposing a more efficient storage format for the dependency representation in Gradle analyzer results. The basic idea is to not repeat the dependency trees for all scopes, but construct an optimized dependency graph, in which dependencies are shared. Ideally, in this graph each dependency occurs once. However, as sometimes packages are referenced from scopes with different sets of dependencies, multiple occurrences are possible.

The most important class of this PR is the new GradleDependencyGraphBuilder class that integrates the project's dependencies into a dependency graph (represented by the DependencyGraph class). The class is currently specific to Gradle, but it should be possible to make it more generic, so that other analyzer implementations can make use of this optimized format as well.

Here are some numbers to give an impression of the size reduction of the dependency representation. The numbers are based on the Mozilla Focus Android project, which was used for testing. This project uses a large number of scopes causing the dependency graph in the old representation to become really huge:

Original representation:

Analyzer execution time: 55:55
Result file size: 1,2 GB
Time to load the result:  111539ms
In-memory representation of analyzer result 1)
byte[]:     45.883.121
String:     45.880.168
Identifier: 11.462.288
PackageRef: 11.461.897

New representation:

Analyzer execution time: 44:55 2)
Result file size: 46 MB
Time to load the result:  5509ms
In-memory representation of analyzer result 1)
byte[]:         36.154
String:         33.200
DependencyRef: 376.067 3)
DependencyIdx:  10.708
Identifier:        892
PackageRef:        501  4)

Remarks:

1. Number of instances of the listed classes, taken from the Memory view of the IntelliJ Debugger.
2. Not sure about the execution time, I actually expected the new approach to take slightly longer because of the additional effort to de-duplicate the dependency graph; however, as this is all done in-memory, it probably does not have a major impact. So the differences in the times could be caused by varying load of our Azure setup.
3. DependencyRef is the counter-part of PackageRef in the new dependency graph representation. It references dependencies by a numeric index rather than an identifier.
4. From the new dependency graph representation the set of Scopes is constructed, but also in an optimized form (shared references to packages). Therefore, the number of PackageRef instances is low.

[oheger-bosch]

By tweaking the representation of dependency references a bit, the size of the result file could be reduced even further to 25.2 MB.

As this approach uses numeric indices to reference dependencies, the result - at least the part with the dependency tree - is now not really readable for human beings. I am not sure whether this is a criterion or not. It would be possible, based on this strongly optimized format, to come to a representation that is still free of most redundancies, but would be more readable (something like a tree of packages that replaces duplicate sub trees by references). But this format would have to be implemented, and it would increase the result size again of course.

[oheger-bosch]

@sschuberth, @mnonnenmacher, @fviernau: It would be great to have a discussion regarding the format of the dependency tree in the Analyzer result. Should it be as efficient as possible (using indices as dependency references) or do you prefer a more human-readable format?

What has been implemented here for Gradle can be applied to other Analyzer implementations as well. I would like to look at Maven next. I have some hope that the de-duplication of dependencies will also improve the performance of this package manager.

[sschuberth]

Hi @oheger-bosch, can you please rebase this PR onto latest master so the Docker image can be built for it out of the box?

[oheger-bosch]

@sschuberth, the PR has been rebased.

[sschuberth]

@mnonnenmacher @fviernau this is looking good to me now, and I'd like to get this merged very soon. While this is not a breaking change in the strict sense, it changes the default output format for new Gradle project analyses. Any objections that I merge this?

[mnonnenmacher]

@mnonnenmacher @fviernau this is looking good to me now, and I'd like to get this merged very soon. While this is not a breaking change in the strict sense, it changes the default output format for new Gradle project analyses. Any objections that I merge this?

Please let me have another look before merging it, I'm currently busy with testing #3564. In the meantime I have a question, do you plan to implement this for the other package managers anytime soon? Archiving two different formats in parallel makes any analysis of the analyzer results very difficult.

[sschuberth]

do you plan to implement this for the other package managers anytime soon?

No, probably not soon. We're planning to do this on-demand, if and and when we run into memory issues with other package managers. But that's purely out of a lack of time and other priorities; otherwise I'd of course like to migrate consistently to the new format as soon as possible.

Archiving two different formats in parallel makes any analysis of the analyzer results very difficult.

Are you referring to using UploadResultToPostgresCommand to put analyzer results into Postgres for gathering statistics? If so, I believe we could rather easily add an optional on-the-fly conversion from new to old format to UploadResultToPostgresCommand.

[mnonnenmacher]

do you plan to implement this for the other package managers anytime soon?

No, probably not soon. We're planning to do this on-demand, if and and when we run into memory issues with other package managers. But that's purely out of a lack of time and other priorities; otherwise I'd of course like to migrate consistently to the new format as soon as possible.

This would also be a chance to align the implementation of the different package managers a bit if the GradleDependencyGraphBuilder is made generic.

Archiving two different formats in parallel makes any analysis of the analyzer results very difficult.

Are you referring to using UploadResultToPostgresCommand to put analyzer results into Postgres for gathering statistics? If so, I believe we could rather easily add an optional on-the-fly conversion from new to old format to UploadResultToPostgresCommand.

Yes, good idea, I have already prepared that change and will make a PR once this is merged.

[sschuberth]

This would also be a chance to align the implementation of the different package managers a bit if the GradleDependencyGraphBuilder is made generic.

Indeed, we should take that opportunity to work a bit towards a more DRY implementation of the package managers which @haikoschol was mentioning a long time ago (I thought we had an issue for that, but I cannot find it anymore).