Order stuck on-hold (arby exits with Unrecoverable error)
#1835
Assignees
Labels
Comments
kilrau
changed the title
Unrecoverable errorUnrecoverable error
|
Interestingly, even after restarting arby, it can't connect to the price stream: https://paste.ubuntu.com/p/sssJG9wsRq/ |
|
After debugging this:
Simnet (correct) Mainnet (missing): |
|
@erkarl @kilrau I have manully built Mainnet is using |
|
Arby was fine for about an hour, then it lost the binance price stream again and did shutdown with unrecoverable error again: Restart fixed it immediately. So in my opinion there is still something wrong with how arby tries to recover the binance price stream since there are scenarios where it can't recover the price stream at run-time but can after restart. |
ghost
assigned
|
Didn't see this again. Closing. |
|
And this one is back again, arby is stuck in this loop: |
Could you please provide more context when this happened? xud was upgraded, right? |
|
Full CLI log: arby-cli.txt You can see arby working fine (orders on both sides - buy/sell in btc/usdt), until suddenly I notice one stuck order on the sell side. Checking arby logs, it's stuck in above loop. If I had to guess it has to do with this one order still being up. |
Fully |
Correct |
I have a feeling this is somehow related to the docker setup and internal DNS as I've not been able to reproduce this outside the docker setup. Mystery 🤔 |
|
Just hit this again: again with a previous arby order "stuck" in xud: |
|
Just had this again and I found that the arby order "stuck" in xud seems to be on-hold. That probably causes arby to fail replacing this order. Alice trying to fill Bob's stuck arby order: Bob trying to remove the stuck arby order: It might be a bug on xud after all. Thoughts? @erkarl @sangaman |
For removing the order you'll need to use the |
|
And again I have a stuck order and arby is in a restart loop due to Restarting xud fixed it and arby issued orders again. In conclusion: I think this order ends up being stuck on-hold (see peer output in above comment signifying that the order is on-hold), thus the order is irremovable and arby errors. Very much looks like an xud issue. Thoughts? @sangaman @erkarl |
Super weird. I'm wondering what triggered xud to put that order on hold? @sangaman |
|
And again. Again one order stuck. |
|
Also, when restarting xud after above, it says it did shutdown gracefully but then just hangs there and waits for docker to be killed: |
|
And one more case leading to this. Arby logs: |
|
Got it too: |
Just a note here that attempting to remove the order by the global id won't work, it's expecting the local id ( |
That is clear and was an oversight on my end, but in #1835 (comment) below you can see the error it gave me when I used the local id correctly. |
kilrau
changed the title
Unrecoverable errorUnrecoverable error)
sangaman
added a commit
that referenced
this issue
Aug 27, 2020
This fixes a bug whereby an order could be placed on hold without that hold ever being removed. When replacing an order, we place the existing order on hold until the replacement order has finished matching, and only then do we remove the original order. However, the order was being placed on hold before the checks for sufficient balance to fill this order, and in case those checks failed the remainder of the `placeOrder` routine would be skipped including the part that removes the original order. In such a case, the original order would be put on hold indefinitely. Instead, we place the existing order on hold immediately before we begin matching and after any checks on the validity of the new order are passed. If the checks fail, then the new order is rejected and the order it was intended to replace remains in the order book without a hold, allowing it to be replaced or removed again by a future call. Fixes #1835.
raladev
pushed a commit
that referenced
this issue
Aug 27, 2020
This fixes a bug whereby an order could be placed on hold without that hold ever being removed. When replacing an order, we place the existing order on hold until the replacement order has finished matching, and only then do we remove the original order. However, the order was being placed on hold before the checks for sufficient balance to fill this order, and in case those checks failed the remainder of the `placeOrder` routine would be skipped including the part that removes the original order. In such a case, the original order would be put on hold indefinitely. Instead, we place the existing order on hold immediately before we begin matching and after any checks on the validity of the new order are passed. If the checks fail, then the new order is rejected and the order it was intended to replace remains in the order book without a hold, allowing it to be replaced or removed again by a future call. Fixes #1835.
|
Just encountered this again with latest xud master.
Since I have one machine where this can be reproduced very reliably (testnet), I'll run reproduce this with xud in log level trace once more and attach logs here. |
|
I'm closing this again because the issue above was not caused by the original issue here, where replacing orders could leave orders stuck on hold forever, but rather because of a hold caused by a "pending" connext payment due to #1850. |
rsercano
pushed a commit
that referenced
this issue
Sep 11, 2020
This fixes a bug whereby an order could be placed on hold without that hold ever being removed. When replacing an order, we place the existing order on hold until the replacement order has finished matching, and only then do we remove the original order. However, the order was being placed on hold before the checks for sufficient balance to fill this order, and in case those checks failed the remainder of the `placeOrder` routine would be skipped including the part that removes the original order. In such a case, the original order would be put on hold indefinitely. Instead, we place the existing order on hold immediately before we begin matching and after any checks on the validity of the new order are passed. If the checks fail, then the new order is rejected and the order it was intended to replace remains in the order book without a hold, allowing it to be replaced or removed again by a future call. Fixes #1835. added currency check to walletwithdraw
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Full logs from 35 mins before crash: https://paste.ubuntu.com/p/kx6vbVDwPD/
The text was updated successfully, but these errors were encountered: