Skip to content

[BOUNTY-2] Add NAT Docker Support #368

Merged
AbdelStark merged 13 commits into
besu-eth:masterfrom
matkt:feature/bounty-add-nat-docker-support
Feb 10, 2020
Merged

[BOUNTY-2] Add NAT Docker Support #368
AbdelStark merged 13 commits into
besu-eth:masterfrom
matkt:feature/bounty-add-nat-docker-support

Conversation

@matkt
Copy link
Copy Markdown
Contributor

@matkt matkt commented Feb 5, 2020

Signed-off-by: Karim TAAM karim.t2am@gmail.com

PR description

The goal of this work is to offer developers a consistent API for performing NAT tasks, and add support for Docker’s NAT implementation when Pantheon is being run from a Docker container.

Here is a proposal for the bounty allowing to detect the Docker environment automatically.

  • Properly detects when Pantheon is being run inside a docker container.

Besu will automatically detect that the client is in a docker.

  • Interacts with Docker APIs, as needed, to determine external IP address and exposed ports.

There is no method to detect the port mapping and the host IP.

There is host.docker.internal which is created by docker and which allows to recover the internal IP used by the host(https://docs.docker.com/docker-for-windows/networking/) but this is not the external IP. In addition we can have several layers between the real external IP address and the besu client (docker, box, etc.)
The port mapping is also not accessible from the container. There are methods but it looks more like a hack than a real solution (https://stackoverflow.com/questions/32444612/how-to-get-the-mapped-port-on-host-from-a-docker-container)

I suggest that if the client detects that it is launched in a docker it will look for :

  • the IP address of a host, given a specific hostname (HOST_IP).
  • the port mapping, given a specific environment variable

So you can run a docker like that :

docker run --add-host=HOST_IP:77.202.XXX.YYY --env HOST_PORT_8545=8046 -it besu-dev:latest --p2p-enabled=true --rpc-http-enabled --rpc-http-apis="eth,net,web3,debug,admin"

matkt added 7 commits January 21, 2020 21:13
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
@AbdelStark AbdelStark requested a review from RatanRSur February 7, 2020 13:19
@AbdelStark AbdelStark added the enhancement New feature or request label Feb 7, 2020
@RatanRSur
Copy link
Copy Markdown
Contributor

While in the process of reviewing it, I made some changes to see if we needed a separate AutoDetectionResult. Take a look at the PR I made against your branch and let me know if you think it's a good change.

I still need to look at the specifics of the Docker advertised port logic and testing.

…upport

Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>

streamline auto detection
@matkt
Copy link
Copy Markdown
Contributor Author

matkt commented Feb 7, 2020

While in the process of reviewing it, I made some changes to see if we needed a separate AutoDetectionResult. Take a look at the PR I made against your branch and let me know if you think it's a good change.

I still need to look at the specifics of the Docker advertised port logic and testing.

LGTM. I validated the modification. Don't hesitate to tell me if you have any questions

Copy link
Copy Markdown
Contributor

@RatanRSur RatanRSur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just had had one question about finding the host ip.

@AbdelStark AbdelStark merged commit 39826b1 into besu-eth:master Feb 10, 2020
@RatanRSur RatanRSur deleted the feature/bounty-add-nat-docker-support branch February 11, 2020 15:24
GregTheGreek pushed a commit to ChainSafe/besu that referenced this pull request Feb 27, 2020
* Plugin error stack traces (besu-eth#369)

Because of how the Log4J2 api works exception stack traces were not
being printed.  Update to use the explicit "throwable" overloaded
methods.

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* VM Trace fixes (besu-eth#372)

* correct refund addresses
* correct returned memory from static precompiled calls.
* update integration test
* precompiles sometimes get plain old CALLs

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* Validate private transaction before sending to enclave (besu-eth#356)

Signed-off-by: Jason Frame <jasonwframe@gmail.com>

* Trace API fixes (besu-eth#377)

- Correct Reporting of reverts in nested call
- correct reporting and handling of value transfer in nested calls
- correct handling of precompiles via DELEGATECALL & CALLCODE
- Addition of precompiled contract gas costs
- Re-work handling of storage writes
- Initial handling of gas refunds
- fix bug in DELEGATECALL tests, we don't need gas in the stack
  * this has a cascading effect on balances in diff tests
- rework depth detection in flat trace
- two new tests blocks

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* [BOUNTY-2] Add NAT Docker Support  (besu-eth#368)

* add docker detection

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* add port mapping detection

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* add tests and refactor ip detection

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* clean RunnerBuilder

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* clean useless modification

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* spotless

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* resolve tests issues

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* streamline auto detection

Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>

Co-authored-by: Abdelhamid Bakhta <45264458+abdelhamidbakhta@users.noreply.github.com>
Co-authored-by: Ratan Rai Sur <ratan.r.sur@gmail.com>

* [PIE-1798] Priv RPC acceptance tests with stub enclave. (besu-eth#330)

* [PIE-1798] Added some Privacy RPC ATs with a stub enclave.

Signed-off-by: Mark Terry <mark.terry@consensys.net>

* More specific task metrics names (besu-eth#389)

A prior refactoring had accidentally removed the specific task names
from the metrics labels.

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* More trace fixes (besu-eth#386)

* pop flat trace context when handling halts
* Better detection of precompiled and non-executed contracts
* correct from address when calling in init code
* fix some exotic nesting cases
* correct from field for init code calls at depth >1
* correct cost on a non-call
* changelog and notes

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* adding the plugin-api javadoc jar at the root level (besu-eth#378)

Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>

* BESU-56: remove erroneous links to errorprone checks (besu-eth#385)

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Changelog entry for multi-tenancy feature (besu-eth#394)

Signed-off-by: Edward Evans <edward.joshua.evans@gmail.com>
Signed-off-by: Jason Frame <jasonwframe@gmail.com>

* 1.5 RC Changelog (besu-eth#395)

* 1.5 RC changelog additions

Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>

* Fixed link (besu-eth#393)

Signed-off-by: Madeline <madeline.murray@consensys.net>

Co-authored-by: Edward <edjojob@gmail.com>

* docker changelog (besu-eth#391)

* docker changelog

Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>

* address comments

Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>

Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com>

* Added known bug to changelog  (besu-eth#388)

* Added known bug

Signed-off-by: Madeline <madeline.murray@consensys.net>

* Added another known bug

Signed-off-by: Madeline <madeline.murray@consensys.net>

* updating version to 1.4.0-rc1 (besu-eth#397)

Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>

* updating verion to 1.4.1-snapshot (besu-eth#398)

Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>

* [BESU-169] cache logs bloom filters automatically. (besu-eth#367)

* First iteration. Draft PR.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* fix SPDX header

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Use block broadcaster to index log bloom.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Remove useless toString method

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* spotless apply

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* cacheLogsBloomForBlockHeader

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* spotless apply

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* ensurePreviousSegmentsArePresent

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Added CLI flag to enable / disable automatic logs bloom indexing.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Create cache directory and cache file if not exist.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Fix acceptance test

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Write cache for block only if block is new canonical head.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Handling of chain reorg.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* fix

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* sportless apply

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Address PR comments.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Remove unused constant.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* spotless apply

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* [BESU-25] Use Devp2p Ping packets at v5 (besu-eth#392)

Broadcast that we support snappy compression

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* Rename logs bloom indexer to log bloom cache to match CLI flag. (besu-eth#401)

Changes class names, variables, and CLI flags as needed.

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* [PIE-1798] Fail cases for multitenancy ATs (besu-eth#400)

Signed-off-by: Mark Terry <mark.terry@consensys.net>

* Private state update metadata and migration (besu-eth#404)

(backport from release-1.4)
Private state update metadata and migration

Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

* [PIE-2303] Automatic log bloom caching - Remove usage of pending file. (besu-eth#407)

* Don't use pending file.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Don't use pending file.

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Create a custom error when plugin is not found. (besu-eth#409)

Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

* Rename method (besu-eth#412)

* rename the method isPersistingState to isPersistingPrivateState because that is what it is used for

Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net>

* rename the method isPersistingState to isPersistingPrivateState because that is what it is used for

Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net>

* rename the method isPersistingState to isPersistingPrivateState because that is what it is used for

Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net>

* [BOUNTY-4] Add NAT Kubernetes Support (besu-eth#410)

* add kubernetes support

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* fix review issues

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* LogBloomCache - make sure the current segment is filled (besu-eth#411)

Make sure we cache the current cache segment with all of the data from
the beginning of the segment.  Use a flip file approach since it will be
 a partial file until done.

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

Co-authored-by: Abdelhamid Bakhta <45264458+abdelhamidbakhta@users.noreply.github.com>

* Reduce recaching in Transaction Log Bloom Filter Cache (besu-eth#415)

Do a cursory cache check at start up (file is present and correct size)
instead of re-generating the cache at startup.

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* fix order of nat detector (besu-eth#414)

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

* Update SLOAD_GAS cost to 200 in Aztlan Gas Calculator (#23) (besu-eth#382)

* Update SLOAD_GAS cost to 200 in Aztlan Gas Calculator

Change SLOAD_GAS cost in Aztlan Gas Calculator from 800 to 200 and
update functions that use SLOAD_GAS.

Signed-off-by: edwardmack <ed@edwardmack.com>

* Update SLOAD_GAS cost to 200 in Aztlan Gas Calculator

Change SLOAD_GAS cost in Aztlan Gas Calculator from 800 to 200 and
update functions that use SLOAD_GAS.

Signed-off-by: edwardmack <ed@edwardmack.com>
Signed-off-by: Edward Mack <ed@edwardmack.com>

* remove overrides

removed overrides of calculateStorageCost and
calculateStorageRefundAmount in AztlanGasCalculator because these were
causing aztlan fork not to sync with kotti testnet.

Signed-off-by: Edward Mack <ed@edwardmack.com>

* merge

Signed-off-by: Edward Mack <ed@edwardmack.com>

* Remove unused ExecutorService init/termination (besu-eth#419)

Signed-off-by: Horacio Mijail Anton Quiles <hmijail@gmail.com>

Co-authored-by: CJ Hare <CjHare@users.noreply.github.com>

* Implement Eth/64 (besu-eth#425)

Wire in the fork identifier into the status messages as Eth64.

Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

* Adds priv_getcode (besu-eth#428)

* Adds priv_getcode

Signed-off-by: Joshua Richardson <joshua@richardson.tech>

* BESU-146 - check if success and return errorResponse otherwise (besu-eth#424)

Signed-off-by: Anthony Buckle <anthonybuckle@gmail.com>

Co-authored-by: CJ Hare <CjHare@users.noreply.github.com>

* fixed typos (besu-eth#429)

Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>

Co-authored-by: CJ Hare <CjHare@users.noreply.github.com>

* roll back on ALL-CAPS.md files for TSC proposal (besu-eth#376)

Signed-off-by: Felipe Faraggi <felipefaraggi@gmail.com>

* updating the changelog with 1.4.0 details (besu-eth#431)

Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>

* [BESU-194] Remove max pivot block resets during fast sync (besu-eth#427)

* remove max pivot block resets during fast sync

* increase max retry number and fix test

* change logs in the handleFailure method

* change logs related to suspicious number of retries
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

Co-authored-by: Danno Ferrin <danno.ferrin@shemnon.com>
Co-authored-by: Jason Frame <jasonwframe@gmail.com>
Co-authored-by: Karim T. <karim.t2am@gmail.com>
Co-authored-by: Abdelhamid Bakhta <45264458+abdelhamidbakhta@users.noreply.github.com>
Co-authored-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Co-authored-by: mark-terry <36909937+mark-terry@users.noreply.github.com>
Co-authored-by: Joshua Fernandes <joshua.fernandes@consensys.net>
Co-authored-by: Antoine Toulme <atoulme@users.noreply.github.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com>
Co-authored-by: Edward <edjojob@gmail.com>
Co-authored-by: Lucas Saldanha <lucascrsaldanha@gmail.com>
Co-authored-by: pinges <16143240+pinges@users.noreply.github.com>
Co-authored-by: Horacio Mijail Antón Quiles <4139546+hmijail@users.noreply.github.com>
Co-authored-by: CJ Hare <CjHare@users.noreply.github.com>
Co-authored-by: anthonybuckle <anthonybuckle@gmail.com>
Co-authored-by: Felipe Faraggi <felipefaraggi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants