[E2E] Setup e2e tests on a single chain; add balances query test #1193

p0mvn · 2022-04-05T09:17:00Z

Closes: #1124

Description

The package e2e defines an integration testing suite used for full end-to-end
testing functionality.

The file e2e_suite_test.go defines the testing suite and contains the core
bootstrapping logic that creates a testing environment via Docker containers.
A testing network is created dynamically with 2 test validators.

The file e2e_test.go contains the actual end-to-end integration tests that
utilize the testing suite.

Currently, there is a single test in e2e_test.go to query the balances of a validator.

More tests are to be added in future PRs

Test Run

Proof of successful e2e test run: https://github.com/p0mvn/osmosis/runs/5830443179?check_suite_focus=true

Next Steps

The following can be split into separate tasks / PRs and tackled one by one:

run a second chain by reusing the current setup
add ibc relayer functionality and enable ibc tests
port all tests from:
- https://github.com/cosmos/gaia/tree/main/tests/e2e
- https://github.com/umee-network/umee/tree/main/tests/e2e
identify more e2e tests specific for Osmosis and write up issues to add them

For contributor use:

Targeted PR against correct branch (see CONTRIBUTING.md)
Updated relevant documentation (docs/) or specification (x/<module>/spec/)
Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
Re-reviewed Files changed in the Github PR explorer

codecov-commenter · 2022-04-05T10:24:40Z

Codecov Report

Merging #1193 (e62e836) into main (eed3294) will decrease coverage by 0.00%.
The diff coverage is 23.07%.

@@            Coverage Diff             @@
##             main    #1193      +/-   ##
==========================================
- Coverage   21.03%   21.03%   -0.01%     
==========================================
  Files         196      196              
  Lines       25349    25351       +2     
==========================================
  Hits         5332     5332              
- Misses      19054    19059       +5     
+ Partials      963      960       -3

Impacted Files	Coverage Δ
x/gamm/keeper/pool_service.go	`56.61% <23.07%> (-0.61%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f57184...e62e836. Read the comment docs.

ValarDragon · 2022-04-05T14:17:21Z

Are a lot of the files in the e2e folder copied from somewhere? Can we perhaps note its source either in a README.md or a doc.go in the package?

alexanderbez · 2022-04-05T14:48:52Z

Are a lot of the files in the e2e folder copied from somewhere? Can we perhaps note its source either in a README.md or a doc.go in the package?

It's copied most likely from my original work in either Gaia or Umee. I don't think there's a need to reference the original. Instead, I propose that we really flesh out the doc.go and/or README.md, which is something I can do.

alexanderbez

🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀

alexanderbez · 2022-04-05T14:50:41Z

.github/workflows/test.yml

@@ -37,7 +37,7 @@ jobs:
      - name: Display go version
        run: go version
      - name: Run all tests
-        run: go test -mod=readonly -timeout 30m -coverprofile=coverage.txt -tags='norace' -covermode=atomic `go list ./... | grep -v simapp`
+        run: go test -mod=readonly -timeout 30m -coverprofile=coverage.txt -tags='norace' -covermode=atomic `go list ./... | grep -E -v 'simapp|e2e'`


nit: we should instead encapsulate this in a make target.

See the changes I made here: https://github.com/cosmos/gaia/blob/main/Makefile#L167-L205

I don't think it's too much trouble to do that in this PR. WDYT?

alexanderbez · 2022-04-05T14:51:48Z

go.mod

@@ -29,6 +29,27 @@ require (
 	gopkg.in/yaml.v2 v2.4.0
 )

+require (


Did you run go mod tidy --compat=1.17? Weird, these should be bundled with the existing indirect require block below.

Yeah, definitely looks like something went not-quite-right here

Had to manually move them to the same block and run go mod tidy

alexanderbez · 2022-04-05T14:52:57Z

tests/e2e/README.md

@@ -0,0 +1,13 @@
+# End-to-end Tests


Don't worry @ValarDragon I'll beef up this doc after merge.

niccoloraspa · 2022-04-05T15:11:42Z

Overall LGTM 🚀

I'm actually reverting to the previous Dockerfile to avoid these kind of permissions issues so we don't have to create a new e2e.Dockerfile

alexanderbez · 2022-04-05T15:15:59Z

Overall LGTM 🚀

I'm actually reverting to the previous Dockerfile to avoid these kind of permissions issues so we don't have to create a new e2e.Dockerfile

So the context on why I originally named it e2e.Dockerfile is because before it was just Dockerfile and validators/node operators assumed that was the image they should use for mainnet, which it wasn't because it has (or can have) specific E2E stuff in it, not intended for mainnet.

niccoloraspa · 2022-04-05T15:33:11Z

So the context on why I originally named it e2e.Dockerfile is because before it was just Dockerfile and validators/node operators assumed that was the image they should use for mainnet, which it wasn't because it has (or can have) specific E2E stuff in it, not intended for mainnet.

That makes absolutely sense but in our setup we are just running a working copy of the Dockerfile that currently has permission problems if not run correctly.

Ideally we should try to use the same Dockerfile for prod and e2e but if they will diverge in the future I'm happy with your solution.

alexanderbez · 2022-04-05T15:47:43Z

Sounds good to me!

ValarDragon · 2022-04-05T21:00:21Z

tests/e2e/e2e_setup_test.go

+	s.Require().NoError(cdc.UnmarshalJSON(appGenState[banktypes.ModuleName], &bankGenState))
+
+	bankGenState.DenomMetadata = append(bankGenState.DenomMetadata, banktypes.Metadata{
+		Description: "An example stable token",


Oh wow, I didn't know there was a description field available in the bank module's per-asset metadata

ValarDragon

Didn't review scaffolding in depth. But the test looks great / its pretty exciting if it 'justworks.TM'.

Should we do the makefile change in this PR or a separate one?

p0mvn · 2022-04-05T21:36:59Z

Making the makefile change here and then will merge

p0mvn · 2022-04-05T22:14:44Z

On another look, I think the makefile should be addressed in a separate PR. Otherwise, the scope ends up being too large. I will merge this and address it in a separate PR

* create e2e image and a makefile step to build * progress * e2e tests in ci * use root distroless image and correct volume path * remove chain b references * implement query balances * implement TestQueryBalances * trigger worflow * trigger * test-e2e Makefile step * fmt and sleep if service unavailable * README * restore branches * add changelog entry * exclude e2e from regular tests * -E flag for grep exclusion * grep * go mod tidy --compat=1.17 * manually tidy go.mod (cherry picked from commit 427e667) # Conflicts: # .github/workflows/test.yml # go.mod # go.sum

* create e2e image and a makefile step to build * progress * e2e tests in ci * use root distroless image and correct volume path * remove chain b references * implement query balances * implement TestQueryBalances * trigger worflow * trigger * test-e2e Makefile step * fmt and sleep if service unavailable * README * restore branches * add changelog entry * exclude e2e from regular tests * -E flag for grep exclusion * grep * go mod tidy --compat=1.17 * manually tidy go.mod

* Setup e2e tests on a single chain; add balances query test (#1193) * create e2e image and a makefile step to build * progress * e2e tests in ci * use root distroless image and correct volume path * remove chain b references * implement query balances * implement TestQueryBalances * trigger worflow * trigger * test-e2e Makefile step * fmt and sleep if service unavailable * README * restore branches * add changelog entry * exclude e2e from regular tests * -E flag for grep exclusion * grep * go mod tidy --compat=1.17 * manually tidy go.mod * second e2e chain with expanded test (#1206) * e2e with IBC tx and test (#1216) * second chain with tests * add hermes, comment out balance query for now * eventually * Update Makefile Co-authored-by: Roman <[email protected]> * remove gas fees and unused functions * readded check * Nicco changes to own hermes image * Remove unused hermes.Dockerfile * Use a single Dockerfile for both debug and official image * readd build hermes in makefile * remove hermes * Set correct golang image and use correct debug image tag Co-authored-by: Adam Tucker <[email protected]> Co-authored-by: Roman <[email protected]> Co-authored-by: Niccolo Raspa <[email protected]> * missing e2e ci job * cleanup Makefile and ci workflows (#1203) * cleanup makefile and ci workflows * update changelog * fix sim test * fix makefile and rename docker repo * fix make test-cover (#1219) * refactor: begin modularizing e2e tests in preparation for upgrade (#1293) * begin modularizing e2e test in preparation for upgrade * rename common package to util * move chain related constants from util to chain package * fix genesis.go * exctract initNodes into the genesis package * remove genesis package, move all logic to chain * continue cleaning up chain package and refactoring e2e * store chains in a slice * reuse common cdc from util package * lexicographical reorder of functions in config.go of chain package * clean up names * refactor: implement Dockerized chain initialization in e2e tests (#1330) Closes: #XXX ## What is the purpose of the change This is a follow-up PR to #1293 and builds upon its work. It is part of the e2e test chain upgrade epic #1235 This PR introduces the ability to run chain initialization in a Docker container by running the following: ``` docker run -v < path >:/tmp/osmo-test osmosis-e2e-chain-init:debug --data-dir=/tmp/osmo-test ``` All chain data is placed at the given `< path >` that is mounted as a volume on the container. In addition, this PR introduces documentation about the current state of the e2e tests. ## Brief change log - [pull chain temp folder creation our of chain to e2e package](c175289) - [create image and makefile steps to initialize chain state](de89119) - [allow for running the upgrade initialization in Docker by providing a data dir](dabb683) - [improve abstractions for chain initialization and add README](cf0d8a3) - improve README ## Testing and Verifying - ran e2e tests locally a few times - `make build-e2e-chain-init` - `make docker-build-e2e-chain-init` - `docker run -v /home/roman/cosmos/osmosis/tmp:/tmp/osmo-test osmosis-e2e-chain-init:debug --data-dir=/tmp/osmo-test` All steps worked as desired ## Documentation and Release Note - Does this pull request introduce a new feature or user-facing behavior changes? no - Is a relevant changelog entry added to the `Unreleased` section in `CHANGELOG.md`? no - How is the feature or change documented? improved `tests/e2e/README.md` ## Next Steps The next step is to switch our chain initialization logic in `func (s *IntegrationTestSuite) configureChain(chainId string)` to use Dockertest and the newly introduced container. Then, mount the genesis and configs on the Osmosis containers, against which the e2e tests are executed * Setup e2e tests on a single chain; add balances query test (#1193) * create e2e image and a makefile step to build * progress * e2e tests in ci * use root distroless image and correct volume path * remove chain b references * implement query balances * implement TestQueryBalances * trigger worflow * trigger * test-e2e Makefile step * fmt and sleep if service unavailable * README * restore branches * add changelog entry * exclude e2e from regular tests * -E flag for grep exclusion * grep * go mod tidy --compat=1.17 * manually tidy go.mod Co-authored-by: Adam Tucker <[email protected]> Co-authored-by: Adam Tucker <[email protected]> Co-authored-by: Niccolo Raspa <[email protected]>

p0mvn marked this pull request as ready for review April 5, 2022 10:27

p0mvn requested review from alexanderbez, ValarDragon, czarcas7ic and niccoloraspa April 5, 2022 10:28

alexanderbez approved these changes Apr 5, 2022

View reviewed changes

ValarDragon reviewed Apr 5, 2022

View reviewed changes

ValarDragon approved these changes Apr 5, 2022

View reviewed changes

p0mvn added 15 commits April 5, 2022 21:24

create e2e image and a makefile step to build

4e9a81b

progress

8148e07

e2e tests in ci

acabc1d

use root distroless image and correct volume path

7d5739b

remove chain b references

97c453f

implement query balances

f02e6f7

implement TestQueryBalances

5f7fdab

trigger worflow

e57caba

trigger

0de615c

test-e2e Makefile step

931bcf5

fmt and sleep if service unavailable

08a922e

README

053f68e

restore branches

f837cab

add changelog entry

d73aaca

exclude e2e from regular tests

eca4d63

p0mvn added 3 commits April 5, 2022 21:25

-E flag for grep exclusion

d4bd78f

grep

6451796

go mod tidy --compat=1.17

3c4336b

p0mvn force-pushed the roman/e2e-single-network branch from a34561d to 3c4336b Compare April 5, 2022 21:26

manually tidy go.mod

e62e836

p0mvn merged commit 427e667 into main Apr 5, 2022

p0mvn deleted the roman/e2e-single-network branch April 5, 2022 22:15

This was referenced Apr 5, 2022

[Epic] Testnet / Chain upgrade flow automation (Sprint 6) #1181

Closed

cleanup Makefile and ci workflows #1203

Merged

niccoloraspa mentioned this pull request Apr 12, 2022

[E2E - Upgrade Testing] Add CI pipelines to test upgrades and chain integrity post-upgrade #1235

Closed

p0mvn added the A:backport/v7.x Do not use. backport patches to v7.x branch label Apr 24, 2022

mergify bot mentioned this pull request Apr 24, 2022

Setup e2e tests on a single chain; add balances query test (backport #1193) #1337

Closed

p0mvn mentioned this pull request Apr 24, 2022

backport: e2e tests (#1193) #1338

Merged

8 tasks

p0mvn changed the title ~~Setup e2e tests on a single chain; add balances query test~~ [E2E] Setup e2e tests on a single chain; add balances query test May 4, 2022

p0mvn mentioned this pull request May 4, 2022

[E2E] Create e2e test set up #1416

Closed

4 tasks

github-actions bot mentioned this pull request Mar 15, 2024

Broken Links Detected #7752

Closed

github-actions bot mentioned this pull request May 15, 2024

Broken Links Detected #8269

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[E2E] Setup e2e tests on a single chain; add balances query test #1193

[E2E] Setup e2e tests on a single chain; add balances query test #1193

p0mvn commented Apr 5, 2022 •

edited

Loading

codecov-commenter commented Apr 5, 2022 •

edited

Loading

ValarDragon commented Apr 5, 2022

alexanderbez commented Apr 5, 2022

alexanderbez left a comment

alexanderbez Apr 5, 2022

alexanderbez Apr 5, 2022

ValarDragon Apr 5, 2022

p0mvn Apr 5, 2022

alexanderbez Apr 5, 2022

niccoloraspa commented Apr 5, 2022

alexanderbez commented Apr 5, 2022

niccoloraspa commented Apr 5, 2022

alexanderbez commented Apr 5, 2022

ValarDragon Apr 5, 2022 •

edited

Loading

ValarDragon left a comment •

edited

Loading

p0mvn commented Apr 5, 2022

p0mvn commented Apr 5, 2022

[E2E] Setup e2e tests on a single chain; add balances query test #1193

[E2E] Setup e2e tests on a single chain; add balances query test #1193

Conversation

p0mvn commented Apr 5, 2022 • edited Loading

Description

Test Run

Next Steps

codecov-commenter commented Apr 5, 2022 • edited Loading

Codecov Report

ValarDragon commented Apr 5, 2022

alexanderbez commented Apr 5, 2022

alexanderbez left a comment

Choose a reason for hiding this comment

alexanderbez Apr 5, 2022

Choose a reason for hiding this comment

alexanderbez Apr 5, 2022

Choose a reason for hiding this comment

ValarDragon Apr 5, 2022

Choose a reason for hiding this comment

p0mvn Apr 5, 2022

Choose a reason for hiding this comment

alexanderbez Apr 5, 2022

Choose a reason for hiding this comment

niccoloraspa commented Apr 5, 2022

alexanderbez commented Apr 5, 2022

niccoloraspa commented Apr 5, 2022

alexanderbez commented Apr 5, 2022

ValarDragon Apr 5, 2022 • edited Loading

Choose a reason for hiding this comment

ValarDragon left a comment • edited Loading

Choose a reason for hiding this comment

p0mvn commented Apr 5, 2022

p0mvn commented Apr 5, 2022

p0mvn commented Apr 5, 2022 •

edited

Loading

codecov-commenter commented Apr 5, 2022 •

edited

Loading

ValarDragon Apr 5, 2022 •

edited

Loading

ValarDragon left a comment •

edited

Loading