activate Reproducible Builds #1337
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hboutemy
requested review from
pierredebelen,
davidharte-gs,
Yasirmod17,
aormerod-gs,
gs-ssh16,
epsstan,
kevin-m-knight-gs,
rafaelbey,
aziemchawdhary-gs,
elopezcastro and
MauricioUyaguari
as code owners
hboutemy
added a commit
to hboutemy/open-developer-platform
that referenced
this pull request
Jan 23, 2023
|
the CLA verification workflow seems broken: finos/open-developer-platform#350 says everything is ok, but this does not reflect here please help |
maoo
reviewed
Jan 23, 2023
Thanks for your contribution @hboutemy ! Legend haven't switched yet to EasyCLA, so we still need to onboard contributors in our previous CLA Bot system; we're working on it, should be done by the end of today. |
|
@cla-bot[bot] check |
|
The cla-bot has been summoned, and re-checked this pull request! |
|
@hboutemy you are all set regarding the CLA check |
Signed-off-by: Hervé Boutemy <[email protected]>
hboutemy
force-pushed
the
reproducible-build
branch
from
0575080
to
5fabd3a
Compare
|
@maoo parent updated, now the update is better scoped |
|
@beekemarie @epsstan @ivan-kyosev-gs - could you please have a look at this PR ? Hervé has kindly worked on making the Legend Engine build "reproducible", you can read more on http://reproducible-builds.org/ ; this is a great opportunity enforce Supply Chain security on Legend, and the idea is to extend this approach across all other Legend repositories. Eager to hear your thoughts. Thanks! |
|
This PR is stale because it has been open for 30 days with no activity. Please remove stale label or add any comment to keep this open. Otherwise this will be closed in 5 days. |
|
@beekemarie @epsstan @ivan-kyosev-gs please review |
kevin-m-knight-gs
approved these changes
Mar 24, 2023
|
Thanks for the contribution! |
ghub-real
pushed a commit
to ghub-real/legend-engine
that referenced
this pull request
Mar 28, 2023
Signed-off-by: Hervé Boutemy <[email protected]> Co-authored-by: Kevin Knight <[email protected]>
rafaelbey
pushed a commit
that referenced
this pull request
Mar 30, 2023
* added the MongoDbExecutionNode * Execution plan processing * mapping model & execution node protocol files. And execution plugin setup (wip) * Execute plan against database * Fix dependency in maven parent pom * Fix dependency in maven parent pom * Fix dependency in maven parent pom * Merge Theo/Hugo's work on execution plan & fix up execution plan module errors * fixed type in extensions * Grammer integration module & compiler module * Grammar integration setup. Compiler via Translator, not working yet - so commented out * Merge MongoDB executor integration with Server * Duplicate dependency warning fixed * Fix diagram & move serializer to the right module * Update version to match upstream master * In Memory mongo setup * fix checksyle warnings * Merge Theo & Hugo's branch for passing in executionPlan from pure to legend-engine. Fixed json _type attribute with custom serializer instead of mixin * Legend java binding setup * replace the _type first character from uppercase to lowercase * Consistent module names * Add GraphExecution node classes * Updated Node specifics interface * Work around maven false positives * Remove local debug setup * Update version numbers to match master * added dependency in monogdb-executionPlan * Fix pom dependency issues & binding codegen * wiring of the mongodb binding extensions in the coreExtensions * checking in these test exectuin plans for reference * graph fetch query can get data from local mongo * Test for json externalize * removed unnecessary dependencies * Remove unused code * Added changes to handle SystemPropertiesSecret credentials Added test class to spin up mongoDB with some test data (Person collection) checked in setup.pure and welcome.pure files * Make firm entity optional and still be able to generate pojo based json input * Json internalize test to adapt to mongo execution result * Fix up dependency errors * removing these * legend-engine-xt-nonrelationalStore-mongodb-executionPlan/pom.xml * Dependency error post Hugo's branch merge, and clean up credentials * Checkstyle warnings * Test setup for demo * Log db responses, and remove json parsing code - as bson already has all the fields necessary * Pass in credentialsgit add legend-engine-xt-nonrelationalStore-mongodb-executionPlan/src/main/java/org/finos/legend/engine/plan/execution/stores/mongodb/MongoDBExecutor.java legend-engine-xt-nonrelationalStore-mongodb-executionPlan/src/main/java/org/finos/legend/engine/plan/execution/stores/mongodb/auth/MongoDBStoreConnectionProvider.java, and remove unused code * Make the function interface correct, to map to concepts * get the mongo query project fields from graphFetch * removed unnecessary pattern * Fix dependency errors * test data consistency * Fix field names * Move query execution code also to test setup * fix: field names to project from the actual field instead of the domain class * Fix testsetup to compile, the extensions parameter as array works only in the IDE, not in compiler * setup legend compile from pure.ide * Binding setup for legend compile * grammar integration wip * Move grammar integration into grammar module * WIP: can stream the graph fetch results back to the PURE IDE * Parser/composer for legend engine * Make server aware of mongodb parser * Add missing pom * empty resources folder * Add antlr gen folder to .ignore * fix inadvertent typo * WIP: can filter on nested properties, working on nested project * Can filter on one level nested property, can create correct project mongo query for nested properties but the PURE IDE can display only top level * merged master and cleanup * Add store compiler + test * Fix name property & remove logging * can get nested properties in PURE IDE, using jackson only for JSON * Compile nested property structure * objectType extends from BaseType, formatting in MongoDBExecutionNodeExecutor * data-space: more analytics improvements (#1570) * data-space: improve specs and analytics * minor cleanups * activate Reproducible Builds (#1337) Signed-off-by: Hervé Boutemy <[email protected]> Co-authored-by: Kevin Knight <[email protected]> * data-space: minor adjustment to analytics (#1571) * [maven-release-plugin] prepare release legend-engine-4.4.5 * [maven-release-plugin] prepare for next development iteration * data-space: improve service executable info (#1573) * [maven-release-plugin] prepare release legend-engine-4.4.6 * [maven-release-plugin] prepare for next development iteration * Legend SQL - introduce pluggable source providers (#1560) * Legend SQL - introduce pluggable source providers --------- Co-authored-by: gs-jp1 <[email protected]> * Legend SQL - further handling of aggregate expressions (#1579) * Persistence Component: Schema Evolution Enhancements and Bug Fixes (#1523) * Support for memsql create statement with shards and column store specification * Support for memsql create statement with shards and column store specification * Support for memsql create statement with shards and column store specification * Support for memsql create statement with shards and column store specification * Support for memsql create statement with shards and column store specification * Support for memsql create statement with shards and column store specification * Implementing statistics collection for Bitemporal Delta milestoning * Implementing statistics collection for Bitemporal Delta milestoning * Implementing statistics collection for Bitemporal Delta milestoning * Implementing statistics collection for Bitemporal Delta milestoning * Implementing configurable infinite batch Id Value and adding ANSI tests for bitemporal milestoning statistics collection * Refactoring assert statements for statistics + changes due to rebase with master * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Add one test * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Add more tests * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Add second passes * Add more tests * Add more tests * Add more tests * Add more tests * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Add more tests * Add more tests * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Adding schema evolution compatibility and data mapping for snowflake + accepting user provided capabilities as a parameter * Construct dataset object from database * Update snowflake data type mapping to handle length and scale * Fix validate primary keys * Update implicit and explicit mappings * Fix bugs for snowflake end-to-end ingest * Fixing h2 end to end tests with schema evolution * Fixing h2 end to end tests with schema evolution * Fix checkstyle * Auto Schema Derivation from Staging * Fix the ingestor class to derive the staging schema * Format the class RelationalIngestor * Add test for Scenario with no fields in main table * Fix all tests * Modify tests for auto deriving schema for main table in H2 * Fix snowflake alter * Clean up method definitions * Clean up tests and TODOs * Change snowflake explicit mapping * Introduce DATA_TYPE_SCALE_CHANGE capability * Disable snowflake explicit data type changes * Code for Handling Case in ingest mode * provide method for datasets case conversion * provide method for datasets case conversion + unit tests * Adding case optimization across datasets and ingestMode in SQL and ingestor mode * Upper case handling in visitors * Code refactor * Adding some code for debug * Fix for Upper case tests --------- Co-authored-by: Mythreyi <[email protected]> Co-authored-by: Mythreyi <[email protected]> Co-authored-by: kumuwu <[email protected]> Co-authored-by: Zhang Lizhi <[email protected]> * update version number to match upstream * remove firm field from testSetup * added README and cleaned up the setup for the mongo execute * Add connection grammar * cleanup and updated README with VM options instructions * executing the query through the new legend::execute method - updated the corresponding README * Mapping parser * Connection parser * Update engine version * Connection parser round trip * remove complier module, part of grammar-integration now * Fix checkstyle errors * Fix tests file as we are now excluding ID column by default * Fix tests file as we are now excluding ID column by default * Remove test files * Compiler for connection * Update version number * missing depedency * avoid creating credentialproviderprovider - test code should have been removed earlier * Pull credential provider from execution state rather than store state * Missing constructor * Update version to 4.5.1 * Fix tests to add new code repositories to expected list * Revert removing the dependency version --------- Signed-off-by: Hervé Boutemy <[email protected]> Co-authored-by: Theodosios Malatestas <[email protected]> Co-authored-by: Hugo Goncalves <[email protected]> Co-authored-by: An Phi <[email protected]> Co-authored-by: Hervé Boutemy <[email protected]> Co-authored-by: Kevin Knight <[email protected]> Co-authored-by: FINOS Administrator <[email protected]> Co-authored-by: Vignesh Manickavasagam <[email protected]> Co-authored-by: gs-jp1 <[email protected]> Co-authored-by: prasar-ashutosh <[email protected]> Co-authored-by: Mythreyi <[email protected]> Co-authored-by: Mythreyi <[email protected]> Co-authored-by: kumuwu <[email protected]> Co-authored-by: Zhang Lizhi <[email protected]>
akphi
added a commit
to akphi/legend-engine
that referenced
this pull request
Apr 14, 2023
|
@hboutemy Thank you for your contribution, we notice that after the upgrade, our release time almost doubled: I have downloaded the log and do a diff to compute the time, I realized that for some reason version 3 of We need to rollback for now (#1688) as this gain in time is quite detrimental for us as it translate to actual cost due to usage of large runner. Having said that, we really appreciate your contribution and the value of reproducible builds 🙏 , hopefully someone could find a way to solve the parallelism problem with |
akphi
added a commit
that referenced
this pull request
Apr 14, 2023
|
@akphi thanks for the info: sorry if that caused that issue, I'm as surprised as you |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Hervé Boutemy [email protected]
What type of PR is this?
enhancement: Reproducible Builds
What does this PR do / why is it needed ?
see https://maven.apache.org/guides/mini/guide-reproducible-builds.html
Which issue(s) this PR fixes:
Fixes #
Other notes for reviewers:
Does this PR introduce a user-facing change?