-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: fixed the error handler & improve stability #448
fix: fixed the error handler & improve stability #448
Conversation
I just increased the redlock await Promise.all([
setupUserWebsocket(logger),
setupCandlesWebsocket(logger, symbols),
setupATHCandlesWebsocket(logger, symbols),
setupTickersWebsocket(logger, symbols)
]); All of them use |
One more update: based on the above code, we have to increase the Why?All promises mentioned above have First try, all of them fail because of redlock
Next try, one of them processed the lock, the others fail
Next try, two of them processed the lock, others fail
Next try, all processed the lock successfully
As we can see, Suggested Solutions
Do you agree with me? If you agree let me know which solution you prefer or if you have another idea. Finally, sorry for the long story 😆 , but I am sure this case will happen to someone because I am facing this as well. |
Sorry for the late reply. I've been busy with work and family. Firstly, thank you for your contributions. It's amazing. Let me review your code and I will give you some feedback if required. |
This looks very good and I can see the code will stabilise the bot. 👍 I will try to run it on my server and see how it goes. |
Thanks @chrisleekr for your feedback. It is indeed a valuable feedback. Morever, I pushed a fix for Thanks again! |
Thanks! As always. :) Let me run it on my machine and see how it goes.
To be honest, I am also not sure what retryCount will be best. It would be depending on the machine's spec, I guess. |
Hello @chrisleekr With I will try to read more about Last thing, can we include a fix for #450 and #451 in this PR? I already wrote explanation there and the fix can be easily implemented. Or it's better to create a new PR? |
Whoa... that is a lot of symbols.
@habibalkhabbaz However, I assume the changes will be conflicted with this PR. In that case, put in this PR. |
Haha, I like trying and doing such things.
Done. Thanks! |
|
||
const execute = async (rawLogger, symbol) => { | ||
const logger = rawLogger.child({ jobName: 'trailingTrade' }); | ||
|
||
await errorHandlerWrapper(logger, 'Trailing Trade', async () => { | ||
if ((await cache.hget(`execute-trailing-trade`, symbol)) === 'true') { | ||
// do nothing, there is another execution task running | ||
return; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can set a flag in cache here to mark there was an attempt to execute processing and check for that flag later to reexecute processing again.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I resolved it accidentally.
It looks like @uhliksk suggesting the queue?
If implements the queue, it would need to check whether should re-queue or not; otherwise, there could be the situation of queueing the execution multiple times if the bot receives the tick multiple times during the lock.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chrisleekr
Yes, @uhliksk is suggesting use a queue instead of simply do nothing.
While I am worried about the delayed execution because of the queue.
To be honest, I don't know which solution is most suitable.
If you have a better idea, drop it here and I will work on it :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we have changed the bot to use WebSocket, this kind of issue is anticipated. Although, I was expecting the tick happens almost every second. But the reality was some coins are not very active.
One of the best Redis queue tools I've used is called Bull queue - https://github.com/OptimalBits/bull
Easy to implement, there is UI as well.
Since you've added the code to prevent multiple executions, adding a queue can be adding extra stability.
The logic would be like,
- Check if the symbol is current in the execution - https://github.com/chrisleekr/binance-trading-bot/pull/448/files/1bd376ac271322d07df190e5ed8ec2b4c9d6f97d..d5a41033b5d5224386bd802a6a4bfd06be3b1b59#
- If true, then queue to Bull. The Bull has a mechanism that can prevent duplicate jobs by setting ID.
Now I am thinking, it would be much better if adding the execution job to Bull whenever receiving the tick.
In addition, all those manual executions we added when the manual task is initiated from the UI can be added to the queue as well.
Although it would require some refactoring...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the inspiration!
In fact, at the beginning of refactoring the code to use the WebSocket, I use the queue but I thought it is not necessary and nothing critical will happen if I remove it and I removed it.
However, after I read your comment I started to rethink and I found it's better to use it, especially when we trigger manual actions as you said.
It can improve stability and management of actions.
I can work on the changes. Will start with small changes when I have time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@habibalkhabbaz
That is awesome.
Sorry for not being able to help you code. Family first...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chrisleekr
Nothing wrong. Enjoy with your family. We will do what we can do with this bot :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for understanding. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking, if you are going to implement the queue, let's merge this PR and make a new PR for the queue.
I think your code is stable.
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello @chrisleekr
Yes, I see it's fine, we can merge now and make a new PR later for the queue.
Cool, I've pulled the change and will run in my machine. |
Description
Based on Node.js documentation, when there is an uncaught exception found using
process.on('uncaughtException')
it's better to solve the actual problem or use try/catch in the right placesReference: https://nodejs.org/api/process.html#warning-using-uncaughtexception-correctly
and based on that, I updated the error handler and used try/catch in the appropriate places to improve the stability.
Motivation and Context
This pull request has improvements based on my last pull request #434, including:
errorHandlerWrapper
and use it in the places where it should beprocess.on('uncaughtException')
and use it only for the missed uncaught exceptions that should be solved (if any) and report them to slackawait
in some places to make try/catch work as expectedgetCacheTrailingTradeSymbols
with an undefined page or less than 1How Has This Been Tested?
I ran the bot for two days and tried to inspect the crashes that may happen or happened and I applied the fixes accordingly.
Test files were also provided.
Thanks