-
Notifications
You must be signed in to change notification settings - Fork 3.9k
Integrating Machine Learning AI instead Technical Analysis to Gekko #789
Comments
I have been thinking about stuff like this for a while, have done small experiments as well. I hope I will have some time in the summer to work on this (will most likely open source this). Semi related to #767 |
This might be relevant https://github.com/buckyroberts/Ella |
Sentiment analysis is definitely a good fit here, and one that I had considered. Another application of machine learning that I am more convinced of is using Genetic Algorithms to both choose the appropriate strategy and then fine tune settings of each strategy for each unique currency pair. We used this approach for tuning machine translation engine settings to produce better output. The big value here is that it could be overlaid on top of any strategy and optimize ROI, including sentiment analysis. Here's a few node based npm modules: |
Just saw the link to #767. Nice! |
Hey all, still after this many links I found an interesting paper from Stanford University. They say that they can predict the daily price change of bitcoin with an accuracy of 98.7% just with random tree forrest. Right now I don't now much about machine learning but i am very interessted into it. Thanks a lot for this great open source project. |
Hi all, first of all nice job on gekko - really good stuff! My take on the AI part is that a solid trading bot should take into consideration all 3 types of analysis. I don't agree that TA is so last century but it may not be enough on it's own, so a combination of:
Would be the best formula using all 3 types of analysis imo. And all 3 need AI to tie them up with historical data and try to make predictions (i.e. getting the twitter feed for the last 6 months, training a NN on the tweets vs price impact and try to predict for future tweets - or perhaps a classification algorithm). However implementing this is non trivial and will require a lot of testing/tuning. I and some friends are considering working on some of these features so perhaps we can share our results if they are worthy. @askmike I wonder what do you think would be the best way of building this sort of functionality? Plugins? Independent projects talking via API? Modules of some sort? I am (mildly) familiar with the codebase, still digging through it. @dramida I like your enthusiasm - you are going for open source with that investment? |
How effective has sentiment analysis been historically? Any evidence? |
afaik there are some attempts in this area but no idea about how effective. I figure it's worth a shot and see what comes out |
I'm starting to implement some self-learning, so it may be worth to try to integrate it with gekko. However, I'm not using js (it would be too hard to do so) and I do use the order book. The idea is that the system learns how to predict the (height of) next peak/drop within a time frame of x times the resolution. Would it be possible to expose order-candles - very similar to the 'trade'-candles - for the calculation of the metrics (in this case, a metric that 'predicts' the height of next peak/drop within a certain timeframe)? |
@kurt-o-sys what do you mean "expose"? It's very easy to run a gekko to simply pipe candles into a database (in semi realtime). If you don't want to use JS you can use whatever you want and pull out of that database (or it's very simple to write a gekko plugin that writes to CSV or whatever you want).
So you want candles exposed so you can process them yourself? Candles are limited to 1 minute but you can take some core components from Gekko that emit trades instead of candles (at this point you are only using func that you can do with ~20 lines of python). |
@klappy i'm agree with you. Genetic algo is more fit to this case. |
@askmike well, I guess I'm misunderstanding something, but I actually want to add indicators, which use trade candles and 'order candles' to calculate the indicator value. The only candles we have now, are the trades. I just want to have the added and removed orders within the ticker time frame as well. |
@kurt-o-sys ah sorry I misunderstood what you meant: When I (in comments & Gekko documentation) talk about candles, I mean a price tick, it's an x minute "summary" of what happened in a markets with regards to price, trades (amount + volume).
I am not sure what you mean by this. |
@askmike OK, let me clarify a little. I expect that Gekko receives 'events', which can be trade or order events. Adding a trade or order does have pretty much the same structure:
Removing an order would be more like:
The candles you're referring to, as mostly done, are, as you say, a summary of what happened in a market, being the summary of the market orders. You could make the same candles for orders: the min/max/open/close/weighted average/quantity. So, one can have OCHL-candles as a summary for the trades (usual market price candles), and one can have OCHL-candles as a summary for the new orders places. (And on top of that, one would need also the orders that are removed.) Another interesting factor could even be the number of trades (or orders). My experience right now is that the order candles, and their dynamics, can give a very accurate price prediction, without even using the trade candles. Combining both, together with a few other factors, makes the price prediction even better. I assume, but still have to test properly, that a well-designed computation graph* are able to detect long-term trends and very short term spikes. I don't use any TA metrics, since these contain less information than 'the dynamics of trades and orders themselves'. The lower the ticker time (I'd prefer something in the range 1-12s **), the better the short term predictions. Does that make any sense, or does it clarify why I like to have the 'order summary of the last tick'? (*) combined neural network, using separate trajectories for orders and trades using memory cells (LSTM), combining them later and adding a few layers, each with their own memory cells. This allows the network to remember the previously calculated values of 'metrics', which can affect the new prediction (or not, depending on the LSTM parameters). (**) I wouldn't save all the data, so I don't need a lot of storage on disk. On each tick, two things happen:
|
@kurt-o-sys when you talk about orders, you are talking about the orderbook (people offering to trade)? In that case: that's out of scope for Gekko. Gekko right now fully ignores order book data to ease backtesting (and sync backtests with live simulations).
They have the same structure in that they both mutate the orderbook, but not in the sense they result in a trade (someone bought BTC vs someone elses USD).
Great! I agree that by watching the orderbook you get a picture better than watching historical trades, the hard thing here is getting this market data.
Yes that makes perfect sense! Though I don't think Gekko is the tool you are looking for, since it is designed to do exactly the opposite: Gekko is designed to be a simple tool that aggregates all historical trades into trade candles (as defined here) and allows you to build TA strategies on top of this data. What you want is something completely different (you want more granual data derived from a different data source (the orderbook), and then you don't want to do anything with TA. I agree that this is a better approach for AI/ML, but it goes directly in the feature set and scope of Gekko. Please read this document: https://gekko.wizb.it/docs/introduction/scope.html (especially the section on market data). So in order to do what you want, what I'd suggest what we do (I think you already started in the previous comments) is:
|
@askmike Allright... too bad. The predictions of next heights/dips, in 4 different time frames (4 different computation graphs), plotted to real data. resolution ticker: 1 minute ~38 times the resolution - black straight lines: max peak within time frame It's somewhat optimized for the dataset itself (which will be the case anyway to some extend in a self-learning system), so in a real situation, I expect it to behave worse. Anyway, I really like the idea and design concept of Gekko a lot. I don't really want the opposite, but more like an extension: adding a metric, but based on orders (which results in the order book) and actual trades. However, since it's out of scope of Gekko, I'll have to build something else around the prediction model :). |
I am working on integrating Machine Learning AI instead Technical Analysis to Gekko. Contact me if you want to help out at [email protected] |
I want to join in , and doing my bachelor project in software . To this project , hit me up @jsappme @askmike ..¨ |
It would be great to integrate some machine learning. Anyone heard of ANN strategy from tradingview? |
@oskarm91, I dont really see where machine learning can be integrated into gekko? And I'm not sure there are plans to integrate GA? Integrating either of them just seems beyond the scope of gekko. As far as GA vs ML vs TA... IMO, GA isn't a replacement for ML, and ML isn't a replacement for TA... but rather all tools that can be used in conjunction to achieve desired results. |
Just been playing around with this idea over the weekend. Im using https://wagenaartje.github.io/neataptic for the neural network The charts are from bitfinex NEOUSD at 15m/3d time frame I feeding it bbands, macd, cci, dema and candlestick information. This is really something ton look into! first train iteration 1000 error 0.025380668695955993 rate 0.01 first evolve iteration 1000 fitness -0.028091147094537194 error 0.021191147094537195 4th evolve iteration 1000x4 fitness -0.02543513253507006 error 0.01863513253507006 |
@FLYBYME very nice thanks for sharing, please contact me at [email protected] |
@FLYBYME Really cool awesome work . Can u also get in contact with me [email protected] |
Very intrested in this subject. I am currently testing an IA (recurrent Neuronal Network) based on the strategy platforme of Gekko. On my point of view, we would need more data and a way of handling it within gekko.
|
@FLYBYME Great. Could you contact me via [email protected] |
@salimelakoui what kind of input data are you using for your network? Do you have a training set that you train the network on? |
@FLYBYME Would you share your code or explain to us how you come to those results? I'm very interested |
@FLYBYME I am working on a bot to catch pump and dumps and profit mad amounts daily. I couldn't find a contact info. I am thinking of merging my bot with AI trading. Please contact me: [email protected] |
Hello,
Inputs are the occurence of indicator's signals (UP DOWN) taken by N
candles.
For exemple :
- RSI-UP and DEMA-UP for day 1
- RSI-UP and DEMA-UP and MACD-UP for day 2
=> It make RSI-UP,DEMA-UP#RSI-UP,DEMA-UP,MACD-UP as an input for 2 candles.
Hence, I calculate all the possible comnbinaisons for that matters and
apply BP considering the % of evolution between the close values.
That is the first net. The second one (which is not a network) would just
identify the best moment to buy considering top and downs between two dates
versus the result of the first network
+ Eviction of certain values that make non sense
+ Calculating result note of first net considering "trust".
I am still stucked
- on identifying "bad moments" where the value is behaving randomly...
- sinification of volumes / trades count ?
Any idea ?
*Elakoui* Directeur Technique
0628730009
[email protected]
Paris
France
[image: Facebook] <https://facebook.com/salim.elakoui> [image: Twitter]
<https://twitter.com/salimelakoui> [image: LinkedIn]
<https://linkedin.com/in/salimelakoui> [image: Google+]
<https://plus.google.com/+SalimElakoui> [image: YouTube]
<https://youtube.com/channel/UCFsNUh_6pp3bLXWMBNbiltQ> [image: Skype] [image:
Flickr] <https://www.flickr.com/photos/salimelakouimartin>
…On Thu, Nov 9, 2017 at 5:19 PM, Tim ***@***.***> wrote:
@salimelakoui <https://github.com/salimelakoui> what kind of input data
are you using for your network? Do you have a training set that you train
the network on?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#789 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ATRgT7b6ofsaPMKqus6xvJBXb_zjE9p1ks5s0yYmgaJpZM4N9gh3>
.
|
Hi, have you ever try with https://github.com/gekkowarez/gekkoga ? |
I've been testing Keras and Tensorflow as the performance I get is better than all the Node libraries I've tested. A big thing one of my engineers pointed out was accounting for transaction times. I'm not entirely sure that the current back testing mechanisms in place do the job. The results that you are getting @FLYBYME is great, I'd love to see live trades and accounting for market Marker/Taker fee's. |
In our research we find it critical to create a data set of successful transactions with corresponding elapsed time to compensate for the network congestion and potential market manipulations. We've seen trade bots like these create losses due to lack of measuring for these circumstances. |
Hello all, hello @dramida, hello @ansonphong, hello @mauroprojetos, |
I have looked into this a little over the past week and I decided to try to implement a trading strategy, that calls a python script every time a new candle arrives. I have a pre-trained network, that simply takes the candle data and is supposed to update the advice using the future values from the network. The problem I have is, that I can't seem to get the interaction between gekko and the python script working. I tried using the python-shell plugin as well as a simple childProcess, both with the same results. If the python script is simply: import sys
print(sys.stdin.read()) Everything works as expected and I can exchange data between node and python. But as soon as I import another module, nothing happens. I believe this has something to do with node's asynchronous nature and the process is killed at some point before it can fully execute. Has anyone tried something similar and succeeded? Or is there a better way to do this? Maybe with the help of a plugin? The strategy file I used looks like this: |
I spent a lot of time on this, and recently. I have some things to report back for anyone interested. I've tried feeding & training an RNN with things like Tech Indicators, raw candle info and both together. I tried different RNN's including Perceptron, LSTM, GRU and NARX. You can achieve all of this without coding outside of node.js or even a trading strategy. Just use neaptaptic.js for your AI and fs to save/load your network. Done. I've had some RNN's work out pretty well. Some of them were regularly outperforming market gains on upswings and still making profits on downswings, including the most recent market corrections. Attaining consistency is challenging and still imperfect, but with the right network training, it's possible. Profitability is definitely attainable. Problems I see: setting hard rules and decision tree logic for an AI to use to make decisions is just another step added to what you would already be doing: setting hard rules with some decision tree logic. If you just feed the network raw data, you'll have to set hard rules on it's output <= which is the same problem. I do see the value in bid/ask prediction, but there is no crystal ball here, we still don't have a window into the future. It's not that hard to set up parameters that change dynamically on a regular trading strategy by doing things like making your macdHistogram buy/sell thresholds a factor of the closing price, and your RSI interval (or any other interval) a factor of: Math.abs(lastMarketRally.dateTime - lastMarketCorrection.dateTime) for example. I did this on a strategy with some other Tech Ind's and I was able to achieve way more without AI including better profit %'s on market upswings and downswings, and most importantly, consistency. I have a degree in finance, so maybe I have an edge up with just the Tech Inds, and a disadvantage on the coding. But my strategy works better than the AI. It's hard for me to argue with results, and weeks of time spent on researching and testing AI. My 2 cents=> If you're new to this, get a good strategy working without AI, then spend time messing around with testing an AI. It will help you get a better understanding of how Technical Indicators work. It'll be like your math teacher forcing you to use a paper and pencil, instead of a calculator for the purpose of understanding the math better. In the end, you may just end up sticking with the paper and pencil, and using the calculator when you need it. One more thing: sentiment analysis and order-book depth were not used in anything I tried. Those may make a significant difference. I will start with adding order-book depth when I feel like picking this up again. You can achieve all of this without coding outside of a strategy as well. Although, it'll be another fs hack, and it'll require running something 24/7 to pull the data. |
I totally agree with you. I am still currently researching/testing/running some strats based on RNN but none of them can successfully predict the future specially hourly and daily. I mostly use AI to reduce my risk. |
When setting up a life trade using config files would they work if I enable 2 or more indicators like rsi and macd and get a nuy sell signal only if both meet set criteria? |
@FLYBYME and/or @salimelakoui could you guys share this project with me I'd like to help out and further this along [email protected] |
I cant code. But I have Ideas. Can a code illiterate individual be of any significant value to the development tech through ideas thought experiments etc? Or should should we just shut up and leave it to the pros? - Since our lack of technical knowledge will eventually lead to more bad than good: Since may not be able to properly determine what is feasible and will probably require extensive time in requesting explanations from people who would be best efforts are probably at writing coding etc.? Or is it possible to determine feasibility of an idea by finding comparable examples in current applications? Asking for a friend... :D |
@Blockchainbuffett I think you should rather bring up such broad topics in tech forums, etc. I'd differentiate between technical knowledge and coding. If you want to have a meaningful conversation in the tech space you need to have some understanding of the topic. If you don't it's doubtful that you'll discover the next big thing before others who do would (or at all). In any case I for one am interested if you feel like reaching out. |
@salimelakoui https://gitlab.nway.org/salim.elakoui/gekko-strategy is not found. Is it public? |
Its nice strat: https://github.com/zschro/gekko-neuralnet/blob/master/strategies/NNv2.js |
Thanks @xFFFFF I will check it out! |
@xFFFFF Do you have an example configuration for NNv2 for USD/ETH which doesnt fail completely? Just to see a better result than -20% (best case so far) when hodling was +20% ;-) |
Fresh, optimized for bnb pairs, candles 10/150 Usdt pairs are hard for most strategies. |
@xFFFFF I tried NNv2 in backtesting but it doesn't do operations.. ideas?
|
Got NNv2 working, but ignore the console window during back test; it's wrong. Haven't found a strat that beats buy and hold yet on 4 years of btc/dash pair Poloniex. @xFFFFF |
Implemented a LSTM RNN using the JS Synpatic library to predict the next closing Bitcoin price. |
Can you take a TA and have a Machine Learning AI built from it? |
Are you getting any results ?
Le sam. 23 juin 2018 à 17:02, Mark Chen <[email protected]> a écrit :
… Implemented a LSTM RNN using the JS Synpatic library to predict the next
closing Bitcoin price.
https://github.com/markchen8717/Gekko_ANN_Strategies/tree/master/Strategies
There's always room for improvement. Open for collab! 👯
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#789 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ATRgT-5Km9HGXHjxbTT_q8016s1LoMLDks5t_lihgaJpZM4N9gh3>
.
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you feel this is very a important issue please reach out the maintainer of this project directly via e-mail: gekko at mvr dot me. |
How many Tech indicators are suitable as imput to NN, and what if the indicator has more then one result. And also, you add the values or just true/false according to buy, sell and the output of neural network will be the predicted price or is better boolean? I messed around it for a long time and still cant say. And also some operating with error autocorelation, make it just adding lagged price to NN or add NN and the rest modell using ARIMA for example? What do you mean. And If you would choose indicators, which will you choose MACD, RSI, BOLL....... and what else seems good and which are worthless. The bad thing is that the input nodes among themselves will show a lot of multicolinearity. |
Yes this is perfect Idea I play with it for some time, but just on the level of neural network, not any combinations. Also I am in the process of choosing best indicators and external imputs, like day of the week, I made decomposition of price series, there is some seasonality which you cannot see, but clear pattern which day are best for buy and sell. So i thing it is another great input. Also may be lagged price of highly collerated currency, but I do not thing in gekko it is possible. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you feel this is very a important issue please reach out the maintainer of this project directly via e-mail: gekko at mvr dot me. |
Predicting stock prices using Technical Analysis is a 16th century Japanese tech.
Let's improve it by creating an Trading Advisor based on Machine Learning who can predict trends.
Using sentiment analysis from Tweeter help also as a second step, as shown in github projects below.
Short technical videos explaining how implement AI in crypto stock prediction:
https://youtu.be/G5Mx7yYdEhE
https://youtu.be/SSu00IRRraY
https://youtu.be/ftMq5ps503w
Here are a list of github projects who resolve the Machine Learning part.
https://github.com/Avhirup/Stock-Market-Prediction-Challenge
https://github.com/ciurana2016/predict_stock_py
http://machinelearningmastery.com/time-series-prediction-with-deep-learning-in-python-with-keras/
I can support the programmer who takes this challenge, financially, like an investment.
The text was updated successfully, but these errors were encountered: