RC-0.6.1

RC-0.6.1

• Fixed RelaysToTokensMultiplier bug
• Added utility CLI command to convert evidence from amino to protobuf
• Updated RPC spec to include stdTx
• Return Dispatch for certain failed relay codes to save a hop on client side
• Fix simulate relay to use basic auth
• Updated User Guide to use RC-0.6.0
• Added unsafe delete command to the keybase
• Added optional flag to bypass interactive prompt for commands that require passphrase input

Upgrade

1. Shutdown Pocket Core (NOTE: Validators will not be jailed until they have been offline for 6 blocks. You have more than one hour to complete this upgrade. 15 to 30 minutes should be sufficient.)

2. !!! Ensure golang version 1.16 !!! or > golang upgrade NOTE: This version is new so please ensure you are on 1.16 or later

   # Startup script linux users might find this useful (BenVan's Validator Script Only)
   sudo apt-get update
   g install 1.16
   

3. Build from source, Homebrew or Docker:

   To build the latest binary from source, follow these steps:

   Navigate into your pocket-core directory:
   Example: cd ~/go/src/github.com/pokt-network/pocket-core

   Enter: pocket version
   You should see: RC-0.5.2.10 (or older)

   To grab the latest packages and tags we are going to swap branches to the latest tag using:

   git pull
   git checkout tags/RC-0.6.1
   Once you checked out the latest tag and branch, we are going to rebuild the binary by entering in:
   go build -o $GOPATH/bin/pocket ./app/cmd/pocket_core/main.go

   After it builds, make sure you are on the latest release version by entering in:
   pocket version

   Output will be RC-0.6.1

   If you built your binary using Homebrew, follow these steps to upgrade your binary:

   In a terminal window, we are going to pull the latest tap by entering:
   $ brew upgrade pokt-network/pocket-core/pocket

   After it builds, make sure you are on the latest version by entering in:
   pocket version

   Output will be RC-0.6.1

   For individuals using Docker, all you will need to do to get the new container image is run:

   docker pull poktnetwork/pocket-core:RC-0.6.1

   or

   docker pull poktnetwork/pocket:RC-0.6.1

   Depending on which of the 2 Docker images you want to use.

4. Upgrade your config.json
   NOTE: There are no config changes from RC-0.6.0
   Use the default config for all options (except unique configurations like moniker, external addr, etc).

   You have two options:
   OPTION 1 (RECOMENDED)

   Run pocket util update-configs command (creates a new config file and backs up old config file)

   In order to use the most performant values, you will need to upgrade your config.json file in within your /config directory.
   The following example assumes your config.json to be ~/.pocket/config, but feel free to swap out your actual with the location in your system.
   Run the pocket util update-configs command to backup your old config and generate a new default one.
   Manually go over your /config/config.json.bk file and update your new /config/config.json with any custom changes that you use.

   OPTION 2

   Backup and remove your existing config.json file (mv ~/.pocket/config/config.json ~/.pocket/config/config.json.bk)

   Execute a CLI command, (this will cause a default config.json to be created), then update the custom configurations.
   Manually go over your datadir/config/config.json.bk file and update your new datadir/config/config.json with any pertinent values such as moniker, external addr, etc.

5. Delete Session.DB before upgrading

   • rm -rf <datadir>/session.db

6. NOTE: Step 6 IS ONLY NEEDED IF RUNNING VERSION EARLIER THAN RC-0.6.0
   Convert pocket_evidence.DB before upgrading

   • pocket util convert-pocket-evidence-db

   Successfully Converted Pocket Evidence DB

   If the command Failed and success message not shown, delete pocket_evidence.DB

7. NOTE: Step 7 IS ONLY NEEDED IF RUNNING VERSION EARLIER RC-0.5.2.9 OR DB CORRUPTED
   GoLevelDB is the only supported database from RC-0.5.2 onward

   • If previously using CLevelDB, users might experience incompatibility issues due to known incompatibilities between the two
   • PNI temporarily will provide a backup datadir to download to avoid syncing from scratch:
     22K .zip
   • After uncompressing theses files, place the contents in the <datadir>/data folder

RC-0.6.0 (UNSTABLE DO NOT USE POST UPGRADE USE 6.1 ^)

RC-0.6.0

Another release, another major milestone.

Presenting RC-0.6.0: Pocket Network's first Consensus Rule Change upgrade.

The development of this release began months before the development of RC-0.5.2.9 and just concluded.

The upcoming Pocket Core release (0.6.0) offers a higher level of security (2 mission critical patches in the merkle tree)
plus provides a higher level of network stability through the removal/patching of events, in addition to a change in the
encoding algorithm (Amino to Google's Protobuf).

Upgrade

1. Shutdown Pocket Core (NOTE: Validators will not be jailed until they have been offline for 6 blocks. You have more than one hour to complete this upgrade. 15 to 30 minutes should be sufficient.)

2. !!! Ensure golang version 1.16 !!! or > golang upgrade NOTE: This version is new so please ensure you are on 1.16 or later

   # Startup script linux users might find this useful
   sudo apt-get update
   g install 1.16
   

3. Build from source, Homebrew or Docker:

   To build the latest binary from source, follow these steps:

   Navigate into your pocket-core directory:
   Example: cd ~/go/src/github.com/pokt-network/pocket-core

   Enter: pocket version
   You should see: RC-0.5.2.10 (or older)

   To grab the latest packages and tags we are going to swap branches to the latest tag using:

   git pull
   git checkout tags/RC-0.6.0
   Once you checked out the latest tag and branch, we are going to rebuild the binary by entering in:
   go build -o $GOPATH/bin/pocket ./app/cmd/pocket_core/main.go

   After it builds, make sure you are on the latest release version by entering in:
   pocket version

   Output will be RC-0.6.0

   If you built your binary using Homebrew, follow these steps to upgrade your binary:

   In a terminal window, we are going to pull the latest tap by entering:
   $ brew upgrade pokt-network/pocket-core/pocket

   After it builds, make sure you are on the latest version by entering in:
   pocket version

   Output will be RC-0.6.0

   For individuals using Docker, all you will need to do to get the new container image is run:

   docker pull poktnetwork/pocket-core:RC-0.6.0

   or

   docker pull poktnetwork/pocket:RC-0.6.0

   Depending on which of the 2 Docker images you want to use.

4. Upgrade your config.json
   Use the default config for all options (except unique configurations like moniker, external addr, etc).

   You have two options:
   OPTION 1 (RECOMENDED)

   Run pocket util update-configs command (creates a new config file and backs up old config file)

   In order to use the most performant values, you will need to upgrade your config.json file in within your /config directory.
   The following example assumes your config.json to be ~/.pocket/config, but feel free to swap out your actual with the location in your system.
   Run the pocket util update-configs command to backup your old config and generate a new default one.
   Manually go over your /config/config.json.bk file and update your new /config/config.json with any custom changes that you use.

   OPTION 2

   Backup and remove your existing config.json file (mv ~/.pocket/config/config.json ~/.pocket/config/config.json.bk)

   Execute a CLI command, (this will cause a default config.json to be created), then update the custom configurations.
   Manually go over your datadir/config/config.json.bk file and update your new datadir/config/config.json with any pertinent values such as moniker, external addr, etc.

5. Delete Session.DB before upgrading

   • rm -rf <datadir>/session.db

6. Delete pocket_evidence.DB before upgrading

   • rm -rf <datadir>/pocket_evidence.db

7. NOTE: Step 7 IS ONLY NEEDED IF RUNNING VERSION EARLIER RC-0.5.2.9 OR DB CORRUPTED
   GoLevelDB is the only supported database from RC-0.5.2 onward

   • If previously using CLevelDB, users might experience incompatibility issues due to known incompatibilities between the two
   • PNI temporarily will provide a backup datadir to download to avoid syncing from scratch:
     22K .zip
   • After uncompressing theses files, place the contents in the <datadir>/data folder

Special Notes

Consensus rule change

Consensus rules are the fabric of the protocol that requires 66% > agreement of Validators in order to reach quorum on the blockchain data.

This edit to the existing Pocket Core Software defines a (dormant) new set of consensus rules that can be activated with a Validator approved governance transaction (See Changelog Below)

This release of Pocket Core supports legacy (RC-0.5) consensus rules as well as the new (RC-0.6) consensus rules.

The software will not change to the new consensus rules until activated by a
66% majority Validator support of a DAO initiated transaction that specifies the height at which the 'rules change'.

Transaction Param (Legacy-Codec)

As described below, the 6.0 upgrade contains a codec upgrade.

Submitting transactions with this release before the upgrade height (will add height here once the DAO votes) will require the 'legacy codec' argument to be TRUE

Submitting transactions with this release after the upgrade height will require the 'legacy codec' argument to be FALSE (DEFAULT)

FAQ

Can I upgrade before of the upgrade height?

Yes, as soon as the release is published. Watch the repo here to be notified when the release is published.

We’ll also make an announcement about the release in the #announcements channel of our discord.

What happens if I do not upgrade in proposed time?

The Pocket Core process will not be able to continue (shutdown automatically and cannot be restarted)

What do I do If I am using a third-party service provider to run my nodes?

If you are using a third-party service provider it will be up to them to upgrade your nodes, but we do recommend that you contact them for their upgrade plan.

We are also coordinating with the major third-party providers directly in order to ensure a smooth upgrade.

What happens if I do not upgrade in time?

Your node will not be able to continue. Full nodes will shut down and be unable to restart without an upgrade. Validators will be slashed and jailed and be unable to restart without an upgrade. If this happens to you, the only way to get your node back online will be to follow the directions outlined in the upgrade guides.

Note: if you manage to recover within 6 blocks of missing the upgrade time, you won’t actually be slashed. But why take that risk?

If I do not upgrade in time, will I lose my POKT?

As explained above, the upgrade force shuts down any node running older versions after the upgrade height.

For Validators, the shutdown can result in standard offline slashing.

Proposal

LINK

Motivation

There are two major security issues in the merkle tree proof/claim implementation as well as an exploitable prediction attack due to a misimplementation at the block hash generation.
The current encoding scheme is both 'custom' and unsupported across most all programming languages which hinders ecosystem growth and future development. Lastly, PUP-4 is somewhat addressed in this release.

Specification

• Convert all consensus level amino encoding (including but not limited to the internal storage codecs) to protobuf encoding while maintaining as many legacy structures as possible

• Introduce Previous Block Validator Voting structure into the block hash used for session and proposer selection algorithms.

• Use the index of the leafs of the plasma core merkle tree as part of the parent hash to lock in the values using the Claim merkle root

• Ensure consensus level events are not concatenated in the pocket core module by initializing in the transaction handler

• Change ABCIValidatorUpdate to ABCIValidatorZeroUpdate for separation of service and validation

Rationale

The bug fixes in the merkle tree result in an increased level of network stability. Applications and node runners will experience an even higher degree of reliability through the new found network security.

Through the addition of Protobuf encoding, client-side tooling such as SDK development and improvements just got a lot easier, which will make expanding our potential app user bases easier and an all around better development experience while using Pocket Network.

In addition a bug identified in event-handling has now been fixed, which creates smaller block sizes and should enable faster txs and overall better service.

We have successfully separated servicing and validation, which allows us to have more nodes overall and more scalability - no longer capped to 5000(technically). That said, PUP4 will likely still try to limit nodes to 5000 due to the lack of jailing available to servicers which may lead to service degradation.

Protobuf encoding will also lower transaction latency a tad because of less resource demand on nodes.

Viability

An extensive number of tests, functional, integration, unit, load, and simulation were completed leading up to this upgrade. The reports will be included in the release notes but we can't release until the DAO has finalized an upgrade height.

Implementation

The implementation of 0.6.0 is near complete. A few pending tests, the agreement of an Upgrade height, and the approval of this proposal, will result in a complete implementation.

Audit

There was no external audit, refer to Viability.

Changelog

• Security patch in merkle sum index (CONSENSUS RULE ...

RC-0.5.2.10

updating default maxInboundPeers to 30. current base config 30 / 10 .

RC-0.5.2.9

RC-0.5.2.9

After nine Beta releases, two Month's worth of continuous internal and external testing, and investigation and QA, Pocket Network's Engineering team feels the resource problems of RC-0.5.0 are fixed (see below for known issues) with the upcoming RC-0.5.2.
Official upgrade guide here

Important Release Notes

1. Delete Session.DB before upgrading from RC-0.5.1

• rm -rf <datadir>/session.db

2. Run this release with the following environment variable: export GODEBUG="madvdontneed=1"
   Link to Golang Issue
3. Use the default config for all options (except unique configurations like moniker, external addr, etc). You have two options:

• Remove/config/config.json file, execute a CLI command, and update the custom configurations
• Run pocket util update-configs command (creates a new config file and backs up old config file)

GoLevelDB is the only supported database from RC-0.5.2 onward

• If previously using CLevelDB, users might experience incompatibility issues due to known incompatibilities between the two
• PNI temporarily will provide a backup datadir to download to avoid syncing from scratch:
  13K .zip
  13K .tar.gz
• After uncompressing theses files, place the contents in the <datadir>/data folder

Context And Original Issues

After a series related issues of Pocket Core's RC-0.5.0 were opened (#1115 #1094 #1116 #1117 ++) in October 2020, PNI opened a formal investigation into the related resource consumption issues of RC-0.5.0 (and subsequently the more stable RC-0.5.1). The main metric of concern with RC-0.5.0 Resources is 'Memory' (virtual, real, RSS, you name it), with a very tangible 'Memory Leak'. 'Relay Stability', though a primary concern for any release, is a secondary concern for RC-0.5.2 as RC-0.5.1 seemed to solve the immediate, emergency level Code 66 errors that plagued blocks 6K-7.5K. Speed is a tertiary concern with RC-0.5.0, taking 10+ hours to sync to Mainnet Block 7000.

Tooling

To debug the issues above, several tools were utilized to determine the root causes of all.

Listed in no particular order:

• Grafana (Observibility/Visibility of resources and consensus issues)
• Google's PProf (CPU and Memory visibility and profile snapshot differences)
• GCVIS (Golang garbage collector monitoring)
• Docker/Docker-Compose (Clean room simulations)
• GCP (Load testing)
• Golang Runtime Pkg (Memstats Testing)
• Golang Debug Pkg (FreeOsMemory Testing)
• GoLand+Debugger (IDE and Debugger)

Debugging and Changelog

Immediately, PNI's team recognized many optimizations to be made within Pocket Core's own source code. This includes the following:

- Delete local Relay/Challenge Evidence on Code 66 failures
- Log relay errors to nodes (don't just return to clients)
- Added configuration to pre-validate auto transactions
- Sending proofs/claims moved to EndBlock
- Load only Blockmeta for PrevCtx
- Added configurable cache PrevCtx, Validators, and Applications
- Don't broadcast claims/proofs if syncing
- Spread out claims/proofs between non-session blocks
- Added max claim age configuration for proof submission 
- Reorganized non-consensus breaking code in Relay/Merkle Verify for efficiency before reads from state
- Configuration to remove ABCILogs
- Fixed (pseudo) memory leak in Tendermints RecvPacketMsg()
- Sessions only store addresses and not entire structs
- Only load bare minimum for relay processing
- Add order to AccountTxs query & blockTxsQuery RPC
- Reduce AccountTxsQuery & blockTxsQuery memory footprint

The results were quite significant in both speed and initial resource usage. Subsequently, the following BETA releases targeted bug fixes and small improvements that were a result of the drastic breaking changes from the original Beta.

- Nondeterministic hash fix
- Code 89 Fix
- Evidence Seal Fix
- Fixes header.TotalTxs !=
- Fixes header.NumTxs !=
- Updating TM version and Version Number to BETA-0.5.2.3
- Upgraded AccountTxs and BlockTxs to use ReducedTxSearch
- Implemented Reduced TxSearch in Tendermint

Will all of this, the speed and 'Relay Stability' concerns seem to be solved. However, the 'Memory Leak' was not fixed. Transparently, the team was surprised and unsure on how to proceed with tackling the issue. One thing that was clear, more visibility was needed to solve the issue. With the addition some much needed tooling (see above), the hunt was on for the leak culprit. Here's a taste of the testing the team did to hunt down this issue:

- 72 hour simulations in Docker
- Clean Room Relay Stress Tests in GCP
- Mainnet `Validator `and `Full Node` Simulations
- Snapshot comparisons between different versions
- Upgrade Path (0.5.1-0.5.2) simulations
- And Much Much More XD

With the help of some close partners and community members, memory offenders were checked off the list:

- Moved IAVL from Tendermint to Pocket Core
- Call LazyLoadVersion/Store for all queries and PrevCtx()
- Reduced Tendermint P2P EnsurePeers actions to prevent leak
- Lowered P2P config to far more conservative numbers
- Updated FastSync to default to V1
- Exposed default leveldb options
- Switched to only go-leveldb for leak benchmarking/performance reasons
- Child process to run madvdontneed if not set
- Updated P2P configs
- fixed nil txIndexer bug (Tendermint now sets txindexer and blockstore)
- removed event type and used Tendermint's abci.Event

Finally, in Beta-0.5.2.8, memory seemed to be at a constant rate.

Evidence

IAVL ISSUES

Memory Bump during a block

IAVL NODE CLONE

Append Events

Tendermint True Bit Indicies

Multiple GCVIS heap stability at Beta 5.2.7

Evidence of cache growth from mempool

External Reports from community members

Disclaimer

Though, the memory seems to be both significantly decreased and stabilized, the team is still not convinced the memory growth issue is fully fixed (though not supported with evidence currently). The team expects to dive deeper and provide even more visibility into Tendermint and Pocket Core in future releases.

Beta-0.5.2.8

Beta-0.5.2.8

NOTE: Please run this release with the following environment variable: export GODEBUG="madvdontneed=1"
NOTE: Please use the default config for all options (except unique configurations like moniker, external addr, etc). To get the default config, remove /config/config.json file and then execute a CLI command)

• fixed nil txIndexer bug (Tendermint now sets txindexer and blockstore)
• removed event type and used Tendermint's abci.Event

Beta-0.5.2.7

NOTE: Please run this release with the following environment variable: export GODEBUG="madvdontneed=1"
NOTE: Please use the default config for all options (where applicable excl. moniker, external addr, etc.)

• Exposed default leveldb options
• Switched to only go-leveldb for leak benchmarking/performance reasons
• Child process to run madvdontneed if not set
• Updated P2P configs

Beta-0.5.2.6

• Reduced Tendermint P2P EnsurePeers actions to prevent leak
• Lowered P2P config to far more conservative numbers
• Updated FastSync to default to V1

Beta-0.5.2.5

• Moved IAVL from Tendermint to Pocket Core
• Call LazyLoadVersion/Store for all queries and PrevCtx()

Beta-0.5.2.4

• Upgraded AccountTxs and BlockTxs to use ReducedTxSearch
• Implemented Reduced TxSearch in Tendermint
• Fix for duplicate evidence / Invalid proposal blocks in Tendermint

Beta-0.5.2.3

Updating TM version and Version Number to BETA-0.5.2.3