Timeout test for serve for jormungandr and http-bridge #528
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
KtorZ
approved these changes
Jul 11, 2019
We still run all tests on successful merges to 'master'
KtorZ
force-pushed
the
piotr/serve_timeout_test
branch
from
d8e6afd to
9b923f6
Compare
KtorZ
added a commit
that referenced
this pull request
Jan 15, 2020
Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ```
KtorZ
added a commit
that referenced
this pull request
Jan 15, 2020
Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ```
KtorZ
added a commit
that referenced
this pull request
Jan 15, 2020
Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ```
iohk-bors bot
added a commit
that referenced
this pull request
Jan 15, 2020
1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1146 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - 06ee934 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 0938999 extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 8f2b197 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 7d3e281 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - e134d52 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 1577706 revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning # Comments <!-- Additional comments or screenshots to attach if any -->  <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]>
KtorZ
added a commit
that referenced
this pull request
Jan 15, 2020
Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ```
iohk-bors bot
added a commit
that referenced
this pull request
Jan 15, 2020
1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1146 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - 06ee934 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 0938999 extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 8f2b197 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 7d3e281 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - e134d52 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 1577706 revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning # Comments <!-- Additional comments or screenshots to attach if any -->  <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]>
iohk-bors bot
added a commit
that referenced
this pull request
Jan 16, 2020
1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1146 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - 06ee934 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 0938999 extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 8f2b197 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 7d3e281 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - e134d52 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 1577706 revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning # Comments <!-- Additional comments or screenshots to attach if any -->  <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]>
iohk-bors bot
added a commit
that referenced
this pull request
Jan 16, 2020
1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1146 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - 92c2aa0 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 206affc extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 3d9bc77 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 4b06845 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - d196fd1 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 423296b revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning - 5eae1fa re-generate nix machinery - fab6e17 fix database closing error handling - Added some additional log messages - Added an extra exception handler - Swapped worker removal with database closing - Reviewed how the in-memory database got cleaned up to avoid race condition - 9babcc2 use on-disk database in integration tests # Comments <!-- Additional comments or screenshots to attach if any -->  <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]>
iohk-bors bot
added a commit
that referenced
this pull request
Jan 16, 2020
1268: endpoint: force wallet resync r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1146 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - 92c2aa0 extend swagger specification with 'forceResync' As an extreme measure to force re-sync the wallet - 206affc extend API with 'forceResync' operation - Added a 'forceResync' link - Extended the servant Client - Provide an initial straightforward implementation re-using 'rollbackBlocks' - 3d9bc77 implement 'forceResync' handler properly Taking care of concurrent workers possibly acting on the database at the same time as well as restarting the restoration worker at the right point - 4b06845 add integration test for the 'forceResync' endpoint Doesn't pass currently because of the in-memory sqlite implementation. It works like a charm with the file database though: ``` (Right before making the request, observe the thread id #142) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.54 UTC] bdb212cc: In sync with the node. (Upon receiving the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:04.97 UTC] [RequestId 10] [DELETE] /v2/wallets/bdb212ccd1d556b62a59f4a4a1de0cff29f61ac1/utxos (Turning off the worker, rolling back, and switching the worker back on) [cardano-wallet.wallet-engine:Info:142] [2020-01-15 10:38:04.97 UTC] bdb212cc: Worker has exited: main action is over. [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:04.98 UTC] bdb212cc: Try rolling back to 0.0 [cardano-wallet.wallet-engine:Info:526] [2020-01-15 10:38:05.00 UTC] bdb212cc: Rolled back to 0.0 (Responding to the request) [cardano-wallet.api-server:Info:526] [2020-01-15 10:38:05.01 UTC] [RequestId 10] 204 No Content in 0.044038928s (And later, the worker restarted, applying blocks from the start, new thread id #528). [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Applying blocks [1136063.1 ... 1144130.1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.03 UTC] bdb212cc: Creating checkpoint at feb1b2c2-[1136063.1#1] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 8920cbb4-[1136063.2#2] [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.04 UTC] bdb212cc: Creating checkpoint at 73a21e9f-[1136063.3#3] [...] (Eventually, reaches the tip) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: MyWallet, created at 2020-01-15 10:35:28.24351865 UTC, not delegating [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: syncProgress: restored [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: discovered 0 new transaction(s) [cardano-wallet.wallet-engine:Info:528] [2020-01-15 10:38:05.18 UTC] bdb212cc: local tip: 8b9aba60-[1144130.1#100] ``` - d196fd1 preserve in-memory database states between calls For the in-memory Sqlite database, we do actually preserve the database after the 'action' is done. This allows for calling 'withDatabase' several times within the same execution and get back the same database. The memory is only cleaned up when calling 'removeDatabase', to mimic the way the file database works! Without this, code like: ```hs Registry.remove re wid registerWorker ctx wid ``` would lead to weird behavior where the wallet `wid` which existed when the worker was stopped didn't exist anymore when register the worker again. This was because, killing the worker closes the connection to the database and re-opening the "same" connection on the in-memory database actually yield a completely different memory representation! - 423296b revise 'expectResponseCode' to give more details on failure It was kinda hard to understand what was going on from the error message. This is now slightly better and at least, allow some reasonning - 5eae1fa re-generate nix machinery - fab6e17 fix database closing error handling - Added some additional log messages - Added an extra exception handler - Swapped worker removal with database closing - Reviewed how the in-memory database got cleaned up to avoid race condition - 9babcc2 use on-disk database in integration tests # Comments <!-- Additional comments or screenshots to attach if any -->  <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue Number
Overview
Comments