feat(geth-all-in-one): add ethereum test image and helper class

[outSH]

• Add new geth-all-in-one test image for running ethereum tests in mainnet-like environment. Image is based on client-go:v1.12.0 and uses Clique (PoS). There is one coinbase account with publicly available keys like in other, similar packages in cacti.
• New image was introduced because currently used open-ethereum one is deprecated.
• Add geth-all-in-one-publish CI for publishing new images.
• Add @hyperledger/cactus-test-geth-ledger for using new geth ledger container in the tests. The class has been moved out of cactus-test-tooling because of conflicting web3js versions. Other than that, it's similar to open-ethereum test class.
• Add basic tests for @hyperledger/cactus-test-geth-ledger. More tests are being developed right now, and should be available in subsequent PRs.

Closes: #2577

[outSH]

@petermetz

• Please publish tools/docker/geth-all-in-one/Dockerfile to hyperledger ghcr so I can replace the references to my repo before merge.
• I've already created issue for adding more tests to this class: https://github.com/hyperledger/cacti/issues/2579 (you can assign it to rwat)
• Side note: As explained in the PR message, I had to add separate package for this class. This may happen in the future too (generic test or common packages decoupled into smaller ones), introducing hard to navigate structure. Have you considered adding additional layer to package organization? Like packages/connectors/X, packages/server/X, etc,, ? Just a rough idea :)

[petermetz]

Please publish tools/docker/geth-all-in-one/Dockerfile to hyperledger ghcr so I can replace the references to my repo before merge.

@outSH Here's the tag I just pushed: ghcr.io/hyperledger/cacti-geth-all-in-one:2023-07-27-2a8c48ed6

[petermetz]

• I've already created issue for adding more tests to this class: test(geth-all-in-one): improve geth helper class tests coverage #2579 (you can assign it to rwat)

@outSH I can't. rwat must comment on it first otherwise GitHub won't let me assign it.

[petermetz]

• Side note: As explained in the PR message, I had to add separate package for this class.

@outSH Yeah, the explanation makes perfect sense, I'll just ask that the new package can be a home to other similar changes in the future as well (same thing could happen to some other ledger's underlying dependencies) which could be achieved by calling it cacti-test-tooling or cacti-test-tooling-2

[petermetz]

• introducing hard to navigate structure. Have you considered adding additional layer to package organization? Like packages/connectors/X, packages/server/X, etc,, ? Just a rough idea :)

@outSH I did, it did come up in the past! The thing is, the way I work on the code, the desired structure is having a single folder ./pkg/ under which all the packages live with their names encoding the hierarchical information that you are wanting to represent with the directory names. My (personal) reasons for preferring the flat directory structure are:

1. It mimics the situation in the package registry well: each package must have a unique name and the npm default is to name your package's directory exactly as the package is called (having the dir and pkg names match exactly is something that objectively reduces confusion while navigating the codebase IMO)
2. Easier to write all sorts of build scripts (though the build scripts ideally would never make assumptions about the directory tree with hard-coded paths like ../../ or ../../../ but this idealistic scenario is rarely feasible unfortunately)
3. Much less bugs in build scripts that go unnoticed for a long time (all of these have happened multiple times and I spent hours figuring them out)
   3.1. tsc output goes in the wrong directory
   3.2. package is not getting matched by lerna/yarn globs that are capturing where the workspace's packages are located - causes breakage in the release scripts
   3.3. Makes the mundane tasks even more mundane: when I have to copy paste something once for each package (something in ci.yaml for example) then if there's varying directory depths/names then I can't quickly copy paste the paths 40 times, I have to stop and think a little bit for every smaller batch of them.
   3.4 post build scripts (such as "build:dev:backend:postbuild": "cp -f ../../yarn.lock ./dist/" get buggy and copy the lock file in the wrong directory)
   3.5. imports break because people don't import correctly via the pkg name and instead do stuff like pulling in modules across packages by walking through the directory structure (e.g. in package A they import X from "../../../../packages/package-b/some-thing";

I know we already have ./examples/ and ./extensions/ and ./packages/ but in my ideal world even all of those would get folded to be just under ./pkg/*

With all that said I know that @VRamakrishna and @sandeepnRES also prefer the hierarchical directories so I'm definitely in the minority here with my preference and therefore we should just make the move to the hierarchical directory structure so I'll ask you the same thing I told them a while back:
Make a proposal in which you assign all [1] packages to a specific directory and then we can all discuss it (and I won't oppose it on the grounds of me wanting the flat structure). It doesn't need to be a PR for now (because a PR like this will suffer from huge merge conflicts as soon as you open it) you can just type it out in the issue description/comments and then we can all come to an agreement and do the pull request during one of the maintainer calls when we have quorum to approve pull requests instantly (and then no trouble with merge conflicts)

[1]

$ ./node_modules/.bin/lerna ls --all
lerna-lite notice cli v1.17.0
check-connection-ethereum-validator                                 (PRIVATE)
@hyperledger/cactus-example-carbon-accounting-backend
@hyperledger/cactus-example-carbon-accounting-business-logic-plugin
@hyperledger/cactus-example-carbon-accounting-frontend
@hyperledger/cactus-example-discounted-asset-trade                  (PRIVATE)
@hyperledger/cactus-example-electricity-trade                       (PRIVATE)
@hyperledger/cactus-example-supply-chain-backend
@hyperledger/cactus-example-supply-chain-business-logic-plugin
@hyperledger/cactus-example-supply-chain-frontend
@hyperledger/cactus-example-tcs-huawei
@hyperledger/cactus-workshop-examples-2022-11-14
@hyperledger/cactus-plugin-htlc-coordinator-besu
@hyperledger/cactus-plugin-object-store-ipfs
@hyperledger/cacti-cmd-gui-app
@hyperledger/cactus-api-client
@hyperledger/cactus-cmd-api-server
@hyperledger/cactus-cmd-socketio-server
@hyperledger/cactus-common
@hyperledger/cactus-core-api
@hyperledger/cactus-core
@hyperledger/cactus-plugin-consortium-manual
@hyperledger/cactus-plugin-htlc-eth-besu-erc20
@hyperledger/cactus-plugin-htlc-eth-besu
@hyperledger/cactus-plugin-keychain-aws-sm
@hyperledger/cactus-plugin-keychain-azure-kv
@hyperledger/cactus-plugin-keychain-google-sm
@hyperledger/cactus-plugin-keychain-memory-wasm
@hyperledger/cactus-plugin-keychain-memory
@hyperledger/cactus-plugin-keychain-vault
@hyperledger/cactus-plugin-ledger-connector-besu
@hyperledger/cactus-plugin-ledger-connector-cdl-socketio
@hyperledger/cactus-plugin-ledger-connector-corda
@hyperledger/cactus-plugin-ledger-connector-fabric-socketio
@hyperledger/cactus-plugin-ledger-connector-fabric
@hyperledger/cactus-plugin-ledger-connector-go-ethereum-socketio
@hyperledger/cactus-plugin-ledger-connector-iroha
@hyperledger/cactus-plugin-ledger-connector-iroha2
@hyperledger/cactus-plugin-ledger-connector-quorum
@hyperledger/cactus-plugin-ledger-connector-sawtooth-socketio
@hyperledger/cactus-plugin-ledger-connector-tcs-huawei-socketio
@hyperledger/cactus-plugin-ledger-connector-ubiquity
@hyperledger/cactus-plugin-ledger-connector-xdai
@hyperledger/cactus-plugin-odap-hermes
@hyperledger/cactus-plugin-persistence-ethereum
@hyperledger/cactus-plugin-persistence-fabric
@hyperledger/cactus-test-api-client
@hyperledger/cactus-test-cmd-api-server
@hyperledger/cactus-test-geth-ledger
@hyperledger/cactus-test-plugin-consortium-manual
@hyperledger/cactus-test-plugin-htlc-eth-besu-erc20
@hyperledger/cactus-test-plugin-htlc-eth-besu
@hyperledger/cactus-test-plugin-ledger-connector-besu
@hyperledger/cactus-test-plugin-ledger-connector-quorum
@hyperledger/cactus-test-tooling
@hyperledger/cactus-test-verifier-client
@hyperledger/cactus-verifier-client
@hyperledger/cacti-weaver-protos-js
@hyperledger/cacti-weaver-driver-fabric
@hyperledger/cacti-weaver-iin-agent
@hyperledger/cacti-weaver-besu-cli
@hyperledger/cacti-weaver-besu-simpleasset
@hyperledger/cacti-weaver-besu-simplestate
@hyperledger/cacti-weaver-fabric-cli
@hyperledger/cacti-weaver-sdk-besu
@hyperledger/cacti-weaver-sdk-fabric
lerna-lite success found 65 packages

[outSH]

@petermetz

Here's the tag I just pushed: ghcr.io/hyperledger/cacti-geth-all-in-one:2023-07-27-2a8c48ed6

Thanks, done!

Yeah, the explanation makes perfect sense, I'll just ask that the new package can be a home to other similar changes in the future as well (same thing could happen to some other ledger's underlying dependencies) which could be achieved by calling it cacti-test-tooling or cacti-test-tooling-2

Sorry, I'm not sure if I understand that correctly :/ Do you want to have two similar packages of cacti-test-tooling? How would anyone keep track of where each feature is located if there's no clear distinction between them? I wanted to keep only this geth ledger class in a separate package (hence the question regarding directory structure). Additional motivation for that is that such class may be useful on it's own in other projects for testing purposes.

[outSH]

I did, it did come up in the past! The thing is, the way I work on the code, the desired structure is having a single folder ./pkg/ under which all the packages live with their names encoding the hierarchical information that you are wanting to

Thanks for such detailed explanation! I'll work on a proposal then :)

[petermetz]

Do you want to have two similar packages of cacti-test-tooling?

Yeah the similarity here is meant to be a feature not a bug, but I recognize that this is highly subjective too, so Ill just dump some more of the details of my reasoning/thoughts and then leave the decision with you and accept whichever way it goes because I also don't want to get into analysis paralysis, we have to keep it the work going as fast as possible instead of me mumbling from some ivory tower of architecture. I already feel bad for typing this much and then you having to read it when we both could've been just writing code instead, so sorry in advance!

1. If I'm looking for a class that is used to pull up a test ledger, I'll know that it must be in one of those two packages. (Ideally of course I would know that it is in the package - the one and only - but that is no longer an option because the web3 upgrade is forcing our hand unfortunately so the next best option is 2 packages IMO)
2. In general I'm usually pretty supportive of adding new packages (because of a bunch of benefits like reducing the blast radius of API changes, reducing the probability of circular dependencies, etc.) but if the package is so specifically named that it only makes sense to have the one class in it then we restrict ourselves from adding other classes that also need a different package because of the same problem. For example if (when) newer versions of Besu/Quorum start us requiring to use web3 v4 as well, then we might find ourselves in the same pickle as now, and then we'll have to add extra packages for all those classes as well (but my intention is that when that happens we could add those classes to this one instead)

How would anyone keep track of where each feature is located if there's no clear distinction between them?

Totally fair point. I don't have a good answer. Maybe we don't name the new one the same, but something just slightly more generic like cacti-test-tooling-web3v4 and that way new versions of the besu, quorum, etc. test ledgers can go in there next to the one you are adding now.

I wanted to keep only this geth ledger class in a separate package (hence the question regarding directory structure).
Additional motivation for that is that such class may be useful on it's own in other projects for testing purposes.

I guess that's my main worry that we are creating a design/architecture precedent where there is a package that has literally the same name as a single class that it contains and in a way committing to making more of these in the future for the other web3v4 test ledgers.

E.g., If a package's purpose is so narrow that it only contains a single class, then we should expand the purpose of the package a little bit (definitely not too much, but a little bit).
Extreme example with hyperbole/exaggeration: if we continue down this path then we could end up with hundreds of packages for our hundreds of classes that we have - it produces the best re-usability but at a cost of much higher complexity, build times, etc.

[jagpreetsinghsasan]

LGTM

[petermetz]

Removing my approval because Jagpreet has also approved and now we are at risk of merging this accidentally without quorum.

@outSH Please re-request review once someone else approved it from a different org as well (e.g. quorum)

[izuru0]

LGTM

[petermetz]

@outSH I see that Izuru has approved it too so now we are ready to go just please do the usual rebase+resolve merge conflicts and pass it back for review!

[outSH]

@petermetz

Yeah the similarity here is meant to be a feature not a bug, but I recognize that this is highly subjective too, so Ill just dump some more of the details of my reasoning/thoughts

Yeah, this is a tricky problem :/

I've found another possible solution, yarn allows installing multiple versions of the same package through aliases - https://classic.yarnpkg.com/lang/en/docs/cli/add/#toc-yarn-add-alias - the docs are for yarn V1 but I've tested it in V2 and it works the same (but it's not documented in a new manual for some reason, so maybe it's not officially supported - we must confirm before use).

As for your previous message, I see the points you mentioned, agree we can easily flood in dozen of tiny packages that will add a lot of headaches and additional complexity that may not be worth it at all in the end. On the other hand, I'm leaning towards feature-based partitioning more that dependency-based one because:

• the latter can become confusing easily too (when there are more conflicting packages for instance).
• devs must know which packages are used in order to quickly locate the test class (v4 for ethereum, v1 for quorum etc..). Can end up with dev using wrong class without knowing that more compatible one is in another package.
• single package can be easily integrated into non-cacti-related projects (this is a long-shot idea to have these clases become general purpose, in separate repo perhaps. Right now it would be more of a burden than benefit for us, but maybe in the future could be considered).

I think it's hard to make this decision now, since it can go bad both ways. I can try merging cactus-test-geth-ledger back to main test package using yarn aliases, or we can keep this package separated and refactor when similar issue occurs in another package (we may have cleaner view of a situation then). I'd prefer the latter since there's no additional work for me now, and test package refactors can be done anytime I think :)

[outSH]

I see that Izuru has approved it too so now we are ready to go just please do the usual rebase+resolve merge conflicts and pass it back for review!

Done

[petermetz]

LGTM