Skip to content
This repository was archived by the owner on Feb 16, 2020. It is now read-only.

Integrating Machine Learning AI instead Technical Analysis to Gekko #789

Closed
dramida opened this issue Jun 18, 2017 · 114 comments
Closed

Integrating Machine Learning AI instead Technical Analysis to Gekko #789

dramida opened this issue Jun 18, 2017 · 114 comments
Labels

Comments

@dramida
Copy link

dramida commented Jun 18, 2017

Predicting stock prices using Technical Analysis is a 16th century Japanese tech.
Let's improve it by creating an Trading Advisor based on Machine Learning who can predict trends.
Using sentiment analysis from Tweeter help also as a second step, as shown in github projects below.
Short technical videos explaining how implement AI in crypto stock prediction:

https://youtu.be/G5Mx7yYdEhE
https://youtu.be/SSu00IRRraY
https://youtu.be/ftMq5ps503w

Here are a list of github projects who resolve the Machine Learning part.

https://github.com/Avhirup/Stock-Market-Prediction-Challenge

https://github.com/ciurana2016/predict_stock_py

http://machinelearningmastery.com/time-series-prediction-with-deep-learning-in-python-with-keras/

I can support the programmer who takes this challenge, financially, like an investment.

@askmike
Copy link
Owner

askmike commented Jun 20, 2017

I have been thinking about stuff like this for a while, have done small experiments as well. I hope I will have some time in the summer to work on this (will most likely open source this).

Semi related to #767

@shanehull
Copy link

This might be relevant https://github.com/buckyroberts/Ella

@klappy
Copy link

klappy commented Jul 28, 2017

Sentiment analysis is definitely a good fit here, and one that I had considered.

Another application of machine learning that I am more convinced of is using Genetic Algorithms to both choose the appropriate strategy and then fine tune settings of each strategy for each unique currency pair. We used this approach for tuning machine translation engine settings to produce better output.

The big value here is that it could be overlaid on top of any strategy and optimize ROI, including sentiment analysis.

Here's a few node based npm modules:

@klappy
Copy link

klappy commented Jul 28, 2017

Just saw the link to #767. Nice!

@musquash
Copy link

musquash commented Aug 14, 2017

Hey all,

still after this many links I found an interesting paper from Stanford University. They say that they can predict the daily price change of bitcoin with an accuracy of 98.7% just with random tree forrest. Right now I don't now much about machine learning but i am very interessted into it.
I hope I can contribute to this project but my programming skills lacks on many parts, too. So I have to learn a lot till I can help here.

Thanks a lot for this great open source project.

@calin-ciobanu
Copy link

calin-ciobanu commented Aug 28, 2017

Hi all, first of all nice job on gekko - really good stuff!

My take on the AI part is that a solid trading bot should take into consideration all 3 types of analysis. I don't agree that TA is so last century but it may not be enough on it's own, so a combination of:

  1. technical analysis (maybe optimized through AI)
  2. fundamental analysis (the stanford paper mentioned by @musquash )
  3. sentiment analysis (mentioned a few times above)

Would be the best formula using all 3 types of analysis imo. And all 3 need AI to tie them up with historical data and try to make predictions (i.e. getting the twitter feed for the last 6 months, training a NN on the tweets vs price impact and try to predict for future tweets - or perhaps a classification algorithm).

However implementing this is non trivial and will require a lot of testing/tuning. I and some friends are considering working on some of these features so perhaps we can share our results if they are worthy.

@askmike I wonder what do you think would be the best way of building this sort of functionality? Plugins? Independent projects talking via API? Modules of some sort? I am (mildly) familiar with the codebase, still digging through it.

@dramida I like your enthusiasm - you are going for open source with that investment?

@thegamecat
Copy link

How effective has sentiment analysis been historically? Any evidence?

@calin-ciobanu
Copy link

afaik there are some attempts in this area but no idea about how effective. I figure it's worth a shot and see what comes out

@kurt-o-sys
Copy link

I'm starting to implement some self-learning, so it may be worth to try to integrate it with gekko. However, I'm not using js (it would be too hard to do so) and I do use the order book. The idea is that the system learns how to predict the (height of) next peak/drop within a time frame of x times the resolution.
For example: if the resolution is 1 minute (I'd prefer to have it lower, but that seems not possible using Gekko right now?), than the system could learn to predict the next peak/drop in the next 12 minutes. Or any other value.

Would it be possible to expose order-candles - very similar to the 'trade'-candles - for the calculation of the metrics (in this case, a metric that 'predicts' the height of next peak/drop within a certain timeframe)?

@askmike
Copy link
Owner

askmike commented Sep 1, 2017

@kurt-o-sys what do you mean "expose"? It's very easy to run a gekko to simply pipe candles into a database (in semi realtime). If you don't want to use JS you can use whatever you want and pull out of that database (or it's very simple to write a gekko plugin that writes to CSV or whatever you want).


Would it be possible to expose order-candles - very similar to the 'trade'-candles - for the calculation of the metrics

So you want candles exposed so you can process them yourself? Candles are limited to 1 minute but you can take some core components from Gekko that emit trades instead of candles (at this point you are only using func that you can do with ~20 lines of python).

@Gebrax
Copy link

Gebrax commented Sep 1, 2017

@klappy i'm agree with you. Genetic algo is more fit to this case.
We can see many pump and dump actions in crypto trading, therefore it's should be hard to predict.
Genetic algo with objective to tune up better stratetegy is what we are looking for.
To get in and out in the right time

@kurt-o-sys
Copy link

kurt-o-sys commented Sep 1, 2017

@askmike well, I guess I'm misunderstanding something, but I actually want to add indicators, which use trade candles and 'order candles' to calculate the indicator value. The only candles we have now, are the trades. I just want to have the added and removed orders within the ticker time frame as well.
Anyway, I'll have to get another look into Gekko, I suppose.

@askmike
Copy link
Owner

askmike commented Sep 1, 2017

@kurt-o-sys ah sorry I misunderstood what you meant:

When I (in comments & Gekko documentation) talk about candles, I mean a price tick, it's an x minute "summary" of what happened in a markets with regards to price, trades (amount + volume).

I just want to have the added and removed orders within the ticker time frame as well.

I am not sure what you mean by this.

@kurt-o-sys
Copy link

kurt-o-sys commented Sep 1, 2017

@askmike OK, let me clarify a little. I expect that Gekko receives 'events', which can be trade or order events. Adding a trade or order does have pretty much the same structure:

type - ADD - market - price (rate) - quantity

Removing an order would be more like:

type - DEL - market - quantity

The candles you're referring to, as mostly done, are, as you say, a summary of what happened in a market, being the summary of the market orders. You could make the same candles for orders: the min/max/open/close/weighted average/quantity. So, one can have OCHL-candles as a summary for the trades (usual market price candles), and one can have OCHL-candles as a summary for the new orders places. (And on top of that, one would need also the orders that are removed.)

Another interesting factor could even be the number of trades (or orders).

My experience right now is that the order candles, and their dynamics, can give a very accurate price prediction, without even using the trade candles. Combining both, together with a few other factors, makes the price prediction even better.

I assume, but still have to test properly, that a well-designed computation graph* are able to detect long-term trends and very short term spikes. I don't use any TA metrics, since these contain less information than 'the dynamics of trades and orders themselves'. The lower the ticker time (I'd prefer something in the range 1-12s **), the better the short term predictions.

Does that make any sense, or does it clarify why I like to have the 'order summary of the last tick'?


(*) combined neural network, using separate trajectories for orders and trades using memory cells (LSTM), combining them later and adding a few layers, each with their own memory cells. This allows the network to remember the previously calculated values of 'metrics', which can affect the new prediction (or not, depending on the LSTM parameters).

(**) I wouldn't save all the data, so I don't need a lot of storage on disk. On each tick, two things happen:

  1. make new predictions and supply them as a metric to the trading advisor (however, I would need the 'order candles' and 'remove orders' events to calculate these metrics)
  2. check if the predictions previously made are close enough. If not, do a new training step by changing the weights of the network - this is the self-learning part (there's no test/validation data: this wouldn't allow for self-learning pretty easily, and in case of market predictions, it makes more sense to allow the network to learn new patterns at any time)

@askmike
Copy link
Owner

askmike commented Sep 1, 2017

@kurt-o-sys when you talk about orders, you are talking about the orderbook (people offering to trade)? In that case: that's out of scope for Gekko. Gekko right now fully ignores order book data to ease backtesting (and sync backtests with live simulations).

Adding a trade or order does have pretty much the same structure:

They have the same structure in that they both mutate the orderbook, but not in the sense they result in a trade (someone bought BTC vs someone elses USD).

My experience right now is that the order candles, and their dynamics, can give a very accurate price prediction, without even using the trade candles. Combining both, together with a few other factors, makes the price prediction even better.

Great! I agree that by watching the orderbook you get a picture better than watching historical trades, the hard thing here is getting this market data.

I assume, but still have to test properly, that a well-designed computation graph* are able to detect long-term trends and very short term spikes. I don't use any TA metrics, since these contain less information than 'the dynamics of trades and orders themselves'. The lower the ticker time (I'd prefer something in the range 1-12s **), the better the short term predictions.

Does that make any sense, or does it clarify why I like to have the 'order summary of the last tick'?

Yes that makes perfect sense! Though I don't think Gekko is the tool you are looking for, since it is designed to do exactly the opposite:

Gekko is designed to be a simple tool that aggregates all historical trades into trade candles (as defined here) and allows you to build TA strategies on top of this data. What you want is something completely different (you want more granual data derived from a different data source (the orderbook), and then you don't want to do anything with TA. I agree that this is a better approach for AI/ML, but it goes directly in the feature set and scope of Gekko. Please read this document:

https://gekko.wizb.it/docs/introduction/scope.html (especially the section on market data).


So in order to do what you want, what I'd suggest what we do (I think you already started in the previous comments) is:

  • define data (+gruanality) needed to train a model.
    • see how to fetch in realtime
    • see how to get historical data (I guess you want to train a model on months of data without waiting for months) <- this is the hard part if you want to use OB data.
  • compare to what gekko is doing and figure out how/if we can integrate.

@kurt-o-sys
Copy link

kurt-o-sys commented Sep 1, 2017

@askmike Allright... too bad.
Just to give you an idea of what I'm talking about:

image

The predictions of next heights/dips, in 4 different time frames (4 different computation graphs), plotted to real data.

resolution ticker: 1 minute

~38 times the resolution - black
~144 times the resolution - green
~ 500 times the resolution - blue
~ 1728 times the resolution - orange

straight lines: max peak within time frame
dashed lines: min dip within time frame

It's somewhat optimized for the dataset itself (which will be the case anyway to some extend in a self-learning system), so in a real situation, I expect it to behave worse.

Anyway, I really like the idea and design concept of Gekko a lot. I don't really want the opposite, but more like an extension: adding a metric, but based on orders (which results in the order book) and actual trades. However, since it's out of scope of Gekko, I'll have to build something else around the prediction model :).

@bitcoinvsalts
Copy link

I am working on integrating Machine Learning AI instead Technical Analysis to Gekko. Contact me if you want to help out at [email protected]

@logicappsource
Copy link

I want to join in , and doing my bachelor project in software . To this project , hit me up @jsappme @askmike ..¨
Mail : [email protected]

@Hagalas
Copy link

Hagalas commented Nov 3, 2017

It would be great to integrate some machine learning.
@thegamecat I don't really get the point of putting genetic algo into gekko 'cause it uses historical data only to select the best parameters for your strategy. I know results can be impressive, especially when you go with the uptrend but does it really that good to play with real money? Wouldn't be better to stick with some neural network instead.

Anyone heard of ANN strategy from tradingview?

@generalectric
Copy link
Contributor

generalectric commented Nov 3, 2017

@oskarm91, I dont really see where machine learning can be integrated into gekko? And I'm not sure there are plans to integrate GA? Integrating either of them just seems beyond the scope of gekko.

As far as GA vs ML vs TA... IMO, GA isn't a replacement for ML, and ML isn't a replacement for TA... but rather all tools that can be used in conjunction to achieve desired results.

@FLYBYME
Copy link

FLYBYME commented Nov 5, 2017

Just been playing around with this idea over the weekend.

Im using https://wagenaartje.github.io/neataptic for the neural network

The charts are from bitfinex NEOUSD at 15m/3d time frame

I feeding it bbands, macd, cci, dema and candlestick information.

This is really something ton look into!

training-data
training-data

first train iteration 1000 error 0.025380668695955993 rate 0.01
first train iteration 1000 error 0 025380668695955993 rate 0 01

first evolve iteration 1000 fitness -0.028091147094537194 error 0.021191147094537195
first evolve iteration 1000 fitness -0 028091147094537194 error 0 021191147094537195

4th evolve iteration 1000x4 fitness -0.02543513253507006 error 0.01863513253507006
4th evolve iteration 1000x4 fitness -0 02543513253507006 error 0 01863513253507006

train iteration 50000 error 0.010022455848223871 rate 0.01
train iteration 50000 error 0 010022455848223871 rate 0 01

@bitcoinvsalts
Copy link

bitcoinvsalts commented Nov 5, 2017

@FLYBYME very nice thanks for sharing, please contact me at [email protected]

@logicappsource
Copy link

@FLYBYME Really cool awesome work . Can u also get in contact with me [email protected]

@salimelakoui
Copy link

Very intrested in this subject.

I am currently testing an IA (recurrent Neuronal Network) based on the strategy platforme of Gekko.

On my point of view, we would need more data and a way of handling it within gekko.

  • RSS / News / Tweets ?
  • PNL ?
  • nosql storage

@John1231983
Copy link

@FLYBYME Great. Could you contact me via [email protected]

@FLYBYME
Copy link

FLYBYME commented Nov 9, 2017

@salimelakoui what kind of input data are you using for your network? Do you have a training set that you train the network on?

@augustoclaro
Copy link

@FLYBYME Would you share your code or explain to us how you come to those results? I'm very interested

@denigada
Copy link

@FLYBYME I am working on a bot to catch pump and dumps and profit mad amounts daily. I couldn't find a contact info. I am thinking of merging my bot with AI trading. Please contact me: [email protected]

@salimelakoui
Copy link

salimelakoui commented Dec 1, 2017 via email

@Filoz
Copy link

Filoz commented Dec 8, 2017

Hi, have you ever try with https://github.com/gekkowarez/gekkoga ?
I'm new and I was googling and documenting about gekko and I found it

@TheRyanHickman
Copy link

I've been testing Keras and Tensorflow as the performance I get is better than all the Node libraries I've tested. A big thing one of my engineers pointed out was accounting for transaction times. I'm not entirely sure that the current back testing mechanisms in place do the job.

The results that you are getting @FLYBYME is great, I'd love to see live trades and accounting for market Marker/Taker fee's.

@TheRyanHickman
Copy link

In our research we find it critical to create a data set of successful transactions with corresponding elapsed time to compensate for the network congestion and potential market manipulations. We've seen trade bots like these create losses due to lack of measuring for these circumstances.

@lutzleonhardt
Copy link

lutzleonhardt commented Feb 15, 2018

Hello all, hello @dramida, hello @ansonphong, hello @mauroprojetos,
I just want to inform about my bitbank-gekko plugin.
Me and @lee101 are looking forward on how to use the prediction data for a good buy/sell strategies (see wiki, just results of a very simple algorithm available).
Help appreciated :)

@chpohl
Copy link

chpohl commented Feb 16, 2018

I have looked into this a little over the past week and I decided to try to implement a trading strategy, that calls a python script every time a new candle arrives. I have a pre-trained network, that simply takes the candle data and is supposed to update the advice using the future values from the network. The problem I have is, that I can't seem to get the interaction between gekko and the python script working. I tried using the python-shell plugin as well as a simple childProcess, both with the same results. If the python script is simply:

import sys
print(sys.stdin.read())

Everything works as expected and I can exchange data between node and python. But as soon as I import another module, nothing happens. I believe this has something to do with node's asynchronous nature and the process is killed at some point before it can fully execute. Has anyone tried something similar and succeeded? Or is there a better way to do this? Maybe with the help of a plugin?

The strategy file I used looks like this:
custom.txt

@jmbartho
Copy link

I spent a lot of time on this, and recently. I have some things to report back for anyone interested.

I've tried feeding & training an RNN with things like Tech Indicators, raw candle info and both together. I tried different RNN's including Perceptron, LSTM, GRU and NARX. You can achieve all of this without coding outside of node.js or even a trading strategy. Just use neaptaptic.js for your AI and fs to save/load your network. Done.

I've had some RNN's work out pretty well. Some of them were regularly outperforming market gains on upswings and still making profits on downswings, including the most recent market corrections. Attaining consistency is challenging and still imperfect, but with the right network training, it's possible. Profitability is definitely attainable.

Problems I see: setting hard rules and decision tree logic for an AI to use to make decisions is just another step added to what you would already be doing: setting hard rules with some decision tree logic. If you just feed the network raw data, you'll have to set hard rules on it's output <= which is the same problem. I do see the value in bid/ask prediction, but there is no crystal ball here, we still don't have a window into the future. It's not that hard to set up parameters that change dynamically on a regular trading strategy by doing things like making your macdHistogram buy/sell thresholds a factor of the closing price, and your RSI interval (or any other interval) a factor of: Math.abs(lastMarketRally.dateTime - lastMarketCorrection.dateTime) for example.

I did this on a strategy with some other Tech Ind's and I was able to achieve way more without AI including better profit %'s on market upswings and downswings, and most importantly, consistency. I have a degree in finance, so maybe I have an edge up with just the Tech Inds, and a disadvantage on the coding. But my strategy works better than the AI. It's hard for me to argue with results, and weeks of time spent on researching and testing AI.

My 2 cents=> If you're new to this, get a good strategy working without AI, then spend time messing around with testing an AI. It will help you get a better understanding of how Technical Indicators work. It'll be like your math teacher forcing you to use a paper and pencil, instead of a calculator for the purpose of understanding the math better. In the end, you may just end up sticking with the paper and pencil, and using the calculator when you need it.

One more thing: sentiment analysis and order-book depth were not used in anything I tried. Those may make a significant difference. I will start with adding order-book depth when I feel like picking this up again. You can achieve all of this without coding outside of a strategy as well. Although, it'll be another fs hack, and it'll require running something 24/7 to pull the data.

@bitcoinvsalts
Copy link

I totally agree with you. I am still currently researching/testing/running some strats based on RNN but none of them can successfully predict the future specially hourly and daily. I mostly use AI to reduce my risk.

@topdev01
Copy link

When setting up a life trade using config files would they work if I enable 2 or more indicators like rsi and macd and get a nuy sell signal only if both meet set criteria?

@clintperry15
Copy link

@FLYBYME and/or @salimelakoui could you guys share this project with me I'd like to help out and further this along [email protected]

@salimelakoui
Copy link

salimelakoui commented Mar 22, 2018 via email

@Blockchainbuffett
Copy link

I cant code. But I have Ideas. Can a code illiterate individual be of any significant value to the development tech through ideas thought experiments etc?

Or should should we just shut up and leave it to the pros? - Since our lack of technical knowledge will eventually lead to more bad than good: Since may not be able to properly determine what is feasible and will probably require extensive time in requesting explanations from people who would be best efforts are probably at writing coding etc.? Or is it possible to determine feasibility of an idea by finding comparable examples in current applications?

Asking for a friend... :D

@thisismydesign
Copy link

@Blockchainbuffett I think you should rather bring up such broad topics in tech forums, etc.

I'd differentiate between technical knowledge and coding. If you want to have a meaningful conversation in the tech space you need to have some understanding of the topic. If you don't it's doubtful that you'll discover the next big thing before others who do would (or at all).

In any case I for one am interested if you feel like reaching out.

@TechupBusiness
Copy link

@salimelakoui https://gitlab.nway.org/salim.elakoui/gekko-strategy is not found. Is it public?

@xFFFFF
Copy link
Contributor

xFFFFF commented May 8, 2018

Its nice strat: https://github.com/zschro/gekko-neuralnet/blob/master/strategies/NNv2.js
Im using this with good results in live.

@TechupBusiness
Copy link

Thanks @xFFFFF I will check it out!

@TechupBusiness
Copy link

@xFFFFF Do you have an example configuration for NNv2 for USD/ETH which doesnt fail completely? Just to see a better result than -20% (best case so far) when hodling was +20% ;-)

@xFFFFF
Copy link
Contributor

xFFFFF commented May 9, 2018

Fresh, optimized for bnb pairs, candles 10/150
SMMA_Length = 12
maFast = 56
maSlow = 368
price_buffer_len = 117
threshold_buy_bear = 6.1
threshold_buy_bull = 1.6
threshold_sell_bear = -0.9
threshold_sell_bull = -1.5

Usdt pairs are hard for most strategies.

@grigio
Copy link

grigio commented May 12, 2018

@xFFFFF I tried NNv2 in backtesting but it doesn't do operations.. ideas?

2018-05-12 22:11:09 (DEBUG):
      Current market: none
      Percent change predicted: -1.0514629247000238
      Current price: 0.023807
      Last buy price: 0
      Current position: none
2018-05-12 22:11:09 (DEBUG):
      Current market: none
      Percent change predicted: -0.012569225190917837
      Current price: 0.023569
      Last buy price: 0
      Current position: none

@bennmann
Copy link

bennmann commented Jun 1, 2018

Got NNv2 working, but ignore the console window during back test; it's wrong. Haven't found a strat that beats buy and hold yet on 4 years of btc/dash pair Poloniex. @xFFFFF

@markchen8717
Copy link

Implemented a LSTM RNN using the JS Synpatic library to predict the next closing Bitcoin price.
https://github.com/markchen8717/Gekko_ANN_Strategies/tree/master/Strategies
There's always room for improvement. Open for collab! 👯

@briancrypto2076
Copy link

Can you take a TA and have a Machine Learning AI built from it?

@salimelakoui
Copy link

salimelakoui commented Aug 26, 2018 via email

@stale
Copy link

stale bot commented Oct 25, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you feel this is very a important issue please reach out the maintainer of this project directly via e-mail: gekko at mvr dot me.

@stale stale bot added the wontfix label Oct 25, 2018
@feterus
Copy link

feterus commented Oct 25, 2018

How many Tech indicators are suitable as imput to NN, and what if the indicator has more then one result. And also, you add the values or just true/false according to buy, sell and the output of neural network will be the predicted price or is better boolean? I messed around it for a long time and still cant say. And also some operating with error autocorelation, make it just adding lagged price to NN or add NN and the rest modell using ARIMA for example? What do you mean. And If you would choose indicators, which will you choose MACD, RSI, BOLL....... and what else seems good and which are worthless. The bad thing is that the input nodes among themselves will show a lot of multicolinearity.

@stale stale bot removed the wontfix label Oct 25, 2018
@feterus
Copy link

feterus commented Oct 25, 2018

view it on GitHub

Are you getting any results ? Le sam. 23 juin 2018 à 17:02, Mark Chen [email protected] a écrit :

Implemented a LSTM RNN using the JS Synpatic library to predict the next closing Bitcoin price. https://github.com/markchen8717/Gekko_ANN_Strategies/tree/master/Strategies There's always room for improvement. Open for collab! 👯 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#789 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/ATRgT-5Km9HGXHjxbTT_q8016s1LoMLDks5t_lihgaJpZM4N9gh3 .

Yes this is perfect Idea I play with it for some time, but just on the level of neural network, not any combinations. Also I am in the process of choosing best indicators and external imputs, like day of the week, I made decomposition of price series, there is some seasonality which you cannot see, but clear pattern which day are best for buy and sell. So i thing it is another great input. Also may be lagged price of highly collerated currency, but I do not thing in gekko it is possible.

@stale
Copy link

stale bot commented Dec 24, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you feel this is very a important issue please reach out the maintainer of this project directly via e-mail: gekko at mvr dot me.

@stale stale bot added the wontfix label Dec 24, 2018
@stale stale bot closed this as completed Dec 31, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests