gan_model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "trying to reproduce this paper\n",
        "\n",
        "1. https://arxiv.org/pdf/2112.03946\n",
        "2. https://github.com/borisbanushev/stockpredictionai\n",
        "3. https://onlinelibrary.wiley.com/doi/10.1155/2018/4907423\n",
        "\n",
        "\n",
        "cannot reproduce this paper. problem is with phase space reconstruction and sliding window. idea is great however I'm not too sure about the results and also there might be data leakages. problem is with using sliding window. how am i supposed to predict testing data . lets say to predict day 1 of testing data we need the previous sliding window size which goes into the size of training data. also utilizing this method problem just gets related to GAN collpase since GANs are very senitive to input data.\n"
      ],
      "metadata": {
        "id": "y4yZj1-y4uBU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FZZtEdLj4e7D"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from tensorflow.keras.layers import Input, LSTM, Dense, Dropout, LeakyReLU, BatchNormalization, Reshape, Conv1D, Flatten\n",
        "from tensorflow.keras.models import Sequential, load_model\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "from tensorflow.keras.regularizers import l1\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "\n",
        "from tensorflow.keras import backend as K\n",
        "\n",
        "\n",
        "\n",
        "# nist represents the amount of stocks we care about\n",
        "nist = 50"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "These following variables below are **hyperparameters** that we should tune after we have a decent model to start with.\n",
        "1. learning rate for the discriminator\n",
        "2. learning rate for the generator\n",
        "3. lambda_p : forcast error loss\n",
        "4. lambda_adv : adversarial loss\n",
        "5. lambda_dpl : direction prediction loss\n",
        "6. batch_size : batch size of the LSTM and CNN\n",
        "7. cnn_lr: the learningrate of the CNN\n",
        "8. strides: the number of strides in the CNN\n",
        "9. lrelu_alpha: the alpha for the LeakyReLU in the GAN\n",
        "10. batchnorm_momentum: momentum for the batch normalisation in the CNN\n",
        "11. padding: the padding in the CNN\n",
        "12. kernel_size':1: kernel size in the CNN\n",
        "13. dropout: dropout in the LSTM\n",
        "14. filters: the initial number of filters\n",
        "15. window_size: the window size for the sliding window\n",
        "16. epochs: the number of epochs for the training\n"
      ],
      "metadata": {
        "id": "EYh4hFCA4__W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# hyperparameters we need to tune for\n",
        "m = 3\n",
        "tau = 1\n",
        "\n",
        "# learning rates\n",
        "learning_rate_d = 0.0001\n",
        "learning_rate_g = 0.0001\n",
        "\n",
        "# coefficients of custom gan loss function\n",
        "lambda_p = 1\n",
        "lambda_adv = 1\n",
        "lambda_dpl = 1\n",
        "\n",
        "batch_size = 32\n",
        "epochs = 500\n",
        "\n",
        "# early stopping for testing. when certain of our model we should remove this\n",
        "patience = 5\n",
        "min_delta = 0.001\n",
        "\n"
      ],
      "metadata": {
        "id": "9GP3-4C54pvh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The data I am currently using is just closing stock prices of 50 different stocks with 500 trading days. It is stored in a column where each column represents a different stock."
      ],
      "metadata": {
        "id": "W-9UHkd_5zcy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data_load = np.loadtxt('prices.txt')\n",
        "\n",
        "column_names = [f'stock_{i+1}' for i in range(nist)]\n",
        "df = pd.DataFrame(data_load, columns=column_names)\n",
        "\n",
        "rows = df.shape[0] - m + 1\n",
        "cols = df.shape[1]\n",
        "\n",
        "# print(rows)\n"
      ],
      "metadata": {
        "id": "F956vFSj5wBx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://github.com/manganganath/stock_price_trend_fft\n",
        "\n",
        "the following fft code is copied from above. furthermore in the paper that I have tried to follow so hard, we use fft to denoise the stock data. All of this is done so that we can normalize the data. Against a lot of current research online, people have commonly used **MIN MAX** to nomralize their data. However I firmly believe that this is wrong. 1. the stock follows a random walk hypotheosis and it doesn't really make sense if we are putting a min max on a stock. we do not know if the stock is going to increase by a certain threshold for perpetuity. It might make sense if we are only looking at returns of a stock who has historically been very stable like banks or etf, but when we want to create a model to predict any stock movement we want to care about these high momentum stocks that could potentially have a lot of change in a short time peroid. These stocks commonly include biotech or tech companies.\n",
        "\n",
        "\n",
        "Many other variation are\n",
        "\n",
        "1. ARIMA\n",
        "2. Wavelet\n",
        "3. FFT\n",
        "\n",
        "to denoise a stock data. However one paper suggested that these perform similar but we can test this later on.\n"
      ],
      "metadata": {
        "id": "-zbLsCoB6nJb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "stock_1 = df['stock_1']\n",
        "close_fft = np.fft.fft(np.asarray(stock_1.tolist()))\n",
        "fft_df = pd.DataFrame({'fft':close_fft})\n",
        "fft_df['absolute'] = fft_df['fft'].apply(lambda x: np.abs(x))\n",
        "fft_df['angle'] = fft_df['fft'].apply(lambda x: np.angle(x))\n",
        "plt.figure(figsize=(14, 7), dpi=100)\n",
        "fft_list = np.asarray(fft_df['fft'].tolist())\n",
        "for num_ in [3, 6, 9, 100]:\n",
        "  fft_list_m10 = np.copy(fft_list); fft_list_m10[num_:-num_]=0\n",
        "  plt.plot(np.fft.ifft(fft_list_m10), label='Fourier transform with {} components'.format(num_))\n",
        "plt.plot(stock_1,  label='Real')\n",
        "plt.xlabel('Days')\n",
        "plt.ylabel('USD')\n",
        "plt.title('stock_1 prices & Fourier transforms')\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "KKK8Am2g6C_S"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is the second part of using phase transformation to normalise the data. In the paper it says that it \"folds the data multiple time and the suggested window size is 131.\" However it did not mention the lag for the phase space reconstruction.\n",
        "\n",
        "Therefore my interpretation of this work is that we use a window size and then we implement phase space reconstruction on each window size for all training data, and then after we predict we use the same window size and the same method to denormalize the data.\n",
        "\n",
        " It shall be noted that using phase space reconstruction is commonly used for determing \"Embedding a time series means reconstructing (approximately) the phase space in which the system evolves in time; in other words, showing how 'all' the variables describing the system evolve in time. By 'all' here we mean more precisely 'the relevant ones'. https://academic.oup.com/book/36525/chapter-abstract/321310592?redirectedFrom=fulltext\".\n",
        "\n",
        " We use it to try and identify \"hidden\" patterns"
      ],
      "metadata": {
        "id": "S6Y1RTi28ipK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def embed_phase_space(data, m, tau):\n",
        "    n = len(data)\n",
        "    embedded_data = np.zeros((n - (m - 1) * tau, m))\n",
        "    for i in range(m):\n",
        "        embedded_data[:, i] = data[i * tau: n - (m - 1) * tau + i * tau]\n",
        "\n",
        "    return embedded_data\n",
        "\n",
        "def normalize_delay_vectors(embedded_data):\n",
        "    norms = np.linalg.norm(embedded_data, axis=1)\n",
        "    normalized_data = embedded_data / norms[:, np.newaxis]\n",
        "    return normalized_data, norms"
      ],
      "metadata": {
        "id": "GP8W4LAb8hDM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# normal_df_list = []\n",
        "\n",
        "def nomralize_data(stock_no, m, tau):\n",
        "    if isinstance(stock_no, (pd.DataFrame, pd.Series)):\n",
        "        stock_no = stock_no.to_numpy()\n",
        "\n",
        "    scaler = StandardScaler()\n",
        "    stock_no_standardized = scaler.fit_transform(stock_no.reshape(-1, 1)).flatten()\n",
        "\n",
        "    close_fft = np.fft.fft(stock_no_standardized)\n",
        "    fft_df = pd.DataFrame({'fft': close_fft})\n",
        "    fft_df['absolute'] = fft_df['fft'].apply(lambda x: np.abs(x))\n",
        "    fft_df['angle'] = fft_df['fft'].apply(lambda x: np.angle(x))\n",
        "\n",
        "    fft_list = np.asarray(fft_df['fft'].tolist())\n",
        "    num_components = 100\n",
        "    fft_list[num_components:-num_components] = 0\n",
        "\n",
        "    reconstructed_signal = np.fft.ifft(fft_list)\n",
        "    reconstructed_signal = reconstructed_signal.real\n",
        "\n",
        "    embedded_data = embed_phase_space(reconstructed_signal, m, tau)\n",
        "\n",
        "    normalized_data, norms = normalize_delay_vectors(embedded_data)\n",
        "\n",
        "    # normal_df = pd.concat([normalized_data, norms, scaler, fft_list], axis=1)\n",
        "    # normal_df.columns = [f'normalized_{j+1}' for j in range(m)] + ['norms', 'scaler', 'fft']\n",
        "    # normal_df_list.append(normal_df)\n",
        "\n",
        "    return normalized_data, norms, scaler, fft_list"
      ],
      "metadata": {
        "id": "N0p_U979_fNs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def denormalize_data(normalized_data, norms, scaler, fft_list):\n",
        "    denormalized_vectors = normalized_data * norms[:, np.newaxis]\n",
        "\n",
        "    reconstructed_signal = np.fft.ifft(fft_list).real\n",
        "\n",
        "    original_data = scaler.inverse_transform(reconstructed_signal.reshape(-1, 1)).flatten()\n",
        "\n",
        "    return original_data"
      ],
      "metadata": {
        "id": "4CsXqxj0_hyO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Look at phase space reconstruction on a piece of data"
      ],
      "metadata": {
        "id": "jt7h0ZiR_m3Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from mpl_toolkits.mplot3d import Axes3D\n",
        "\n",
        "normalized_data, norms, scaler, fft_list = nomralize_data(df['stock_1'], 3, 1)\n",
        "\n",
        "fig = plt.figure(figsize=(10, 7))\n",
        "ax = fig.add_subplot(111, projection='3d')\n",
        "ax.plot(normalized_data[:, 0], normalized_data[:, 1], normalized_data[:, 2], lw=0.5)\n",
        "ax.set_xlabel('X')\n",
        "ax.set_ylabel('Y')\n",
        "ax.set_zlabel('Z')\n",
        "plt.title('Phase Space Reconstruction')\n",
        "plt.show()\n",
        "\n",
        "print(normalized_data.shape)\n",
        "\n",
        "# denormalized_data = denormalize_data(normalized_data, norms, scaler, fft_list)\n",
        "# print(denormalized_data.shape)"
      ],
      "metadata": {
        "id": "KvkNCo6G_lmJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def create_sliding_windows(data, window_size):\n",
        "    windows = []\n",
        "    for i in range(len(data) - window_size + 1):\n",
        "        windows.append(data[i:i + window_size])\n",
        "    return np.array(windows)\n",
        "\n",
        "\n",
        "def create_data_vector(df, window_size):\n",
        "    stock_data = []\n",
        "\n",
        "    for i in range(nist):\n",
        "        stock_series = df[f'stock_{i+1}'].values.flatten()\n",
        "        sliding_windows = create_sliding_windows(stock_series, window_size)\n",
        "        for window in sliding_windows:\n",
        "            normalized_stock_data, norms, scaler, fft_list = nomralize_data(window, m, tau)\n",
        "            stock_data.append(normalized_stock_data)\n",
        "\n",
        "    return np.array(stock_data)"
      ],
      "metadata": {
        "id": "4-lg4xMd8FHO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def adversarial_loss(y_true, y_pred):\n",
        "    return tf.keras.losses.binary_crossentropy(y_true, y_pred)\n",
        "\n",
        "def forecast_error_loss(y_true, y_pred):\n",
        "    return tf.reduce_mean(tf.square(y_true - y_pred))\n",
        "\n",
        "def direction_prediction_loss(y_true, y_pred):\n",
        "    if y_true.shape[0] > 1:\n",
        "        sign_true = tf.sign(y_true[1:] - y_true[:-1])\n",
        "        sign_pred = tf.sign(y_pred[1:] - y_pred[:-1])\n",
        "        return tf.reduce_mean(tf.abs(sign_true - sign_pred))\n",
        "    else:\n",
        "        return 0.0\n",
        "\n",
        "def custom_gan_loss(y_true, y_pred):\n",
        "    adv_loss = adversarial_loss(y_true, y_pred)\n",
        "    p_loss = forecast_error_loss(y_true, y_pred)\n",
        "    dpl_loss = direction_prediction_loss(y_true, y_pred)\n",
        "    return lambda_adv * adv_loss + lambda_p * p_loss + lambda_dpl * dpl_loss\n"
      ],
      "metadata": {
        "id": "U-Xn-mIp8M74"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "A lot of below follows https://github.com/soumith/ganhacks where I hoped this would of address the issue of GAN collpase"
      ],
      "metadata": {
        "id": "vZBqFnyC_1wr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "'''\n",
        "  for our generator we are building an lstm model. right now it is a simple lstm but we can use this later.\n",
        "  https://arxiv.org/pdf/1607.04381v1\n",
        "  shape is 500, 5+m -> 1 which is the price of the stock\n",
        "'''\n",
        "\n",
        "def build_generator(input_shape):\n",
        "    model = Sequential()\n",
        "    model.add(LSTM(input_shape[0], input_shape=input_shape, return_sequences=True, kernel_regularizer=l1(0.01)))\n",
        "    model.add(Dropout(0.2))\n",
        "    model.add(Dense(1, activation='linear'))\n",
        "\n",
        "  # print(model.summary())\n",
        "    return model\n",
        "\n",
        "# input_shape = (rows, 5 + m)\n",
        "# print(input_shape)\n",
        "# build_generator(input_shape)\n"
      ],
      "metadata": {
        "id": "y1Rs7jkK8TzA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "'''\n",
        "Sequential(\n",
        "  (0): Conv1D(None -> 32, kernel_size=(5,), stride=(2,))\n",
        "  (1): LeakyReLU(0.01)\n",
        "  (2): Conv1D(None -> 64, kernel_size=(5,), stride=(2,))\n",
        "  (3): LeakyReLU(0.01)\n",
        "  (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
        "  (5): Conv1D(None -> 128, kernel_size=(5,), stride=(2,))\n",
        "  (6): LeakyReLU(0.01)\n",
        "  (7): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
        "  (8): Dense(None -> 220, linear)\n",
        "  (9): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)\n",
        "  (10): LeakyReLU(0.01)\n",
        "  (11): Dense(None -> 220, linear)\n",
        "  (12): Activation(relu)\n",
        "  (13): Dense(None -> 1, linear)\n",
        ")\n",
        "\n",
        "'''\n",
        "def build_discriminator(input_shape):\n",
        "    model = Sequential()\n",
        "    model.add(Conv1D(32, kernel_size=5, strides=2, input_shape=input_shape, padding='same'))\n",
        "    model.add(LeakyReLU(0.01))\n",
        "    model.add(Conv1D(64, kernel_size=5, strides=2, padding='same'))\n",
        "    model.add(LeakyReLU(0.01))\n",
        "    model.add(BatchNormalization(axis=-1, epsilon=1e-05, momentum=0.9))\n",
        "    model.add(Conv1D(128, kernel_size=5, strides=2, padding='same'))\n",
        "    model.add(LeakyReLU(0.01))\n",
        "    model.add(BatchNormalization(axis=-1, epsilon=1e-05, momentum=0.9))\n",
        "\n",
        "    model.add(Flatten())\n",
        "    model.add(Dense(220, activation='linear'))\n",
        "    model.add(BatchNormalization(axis=-1, epsilon=1e-05, momentum=0.9))\n",
        "    model.add(LeakyReLU(0.01))\n",
        "    model.add(Dense(220, activation='relu'))\n",
        "\n",
        "    model.add(Dense(1, activation='linear'))\n",
        "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "\n",
        "    # print(model.summary())\n",
        "    return model\n",
        "\n",
        "# input_shape = (rows, 5 + m)\n",
        "# build_discriminator(input_shape)\n"
      ],
      "metadata": {
        "id": "5xaKgiSJ_0z2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def train_gan(generator, discriminator, data):\n",
        "    optimizer_g = Adam(learning_rate=learning_rate_g)\n",
        "    optimizer_d = Adam(learning_rate=learning_rate_d)\n",
        "    best_loss = float('inf')\n",
        "    epochs_no_improve = 0\n",
        "\n",
        "\n",
        "    @tf.function\n",
        "    def train_step(real_data):\n",
        "        current_batch_size = tf.shape(real_data)[0]\n",
        "\n",
        "        noise = tf.random.normal([current_batch_size, data.shape[1], data.shape[2]])\n",
        "\n",
        "        with tf.GradientTape() as tape_g, tf.GradientTape() as tape_d:\n",
        "            generated_data = generator(noise, training=True)\n",
        "\n",
        "            real_output = discriminator(real_data, training=True)\n",
        "            fake_output = discriminator(generated_data, training=True)\n",
        "\n",
        "            # custom loss as referenced in the paper\n",
        "            g_loss = custom_gan_loss(tf.ones_like(fake_output), fake_output)\n",
        "\n",
        "            d_loss_real = tf.keras.losses.binary_crossentropy(tf.ones_like(real_output), real_output)\n",
        "            d_loss_fake = tf.keras.losses.binary_crossentropy(tf.zeros_like(fake_output), fake_output)\n",
        "            d_loss = d_loss_real + d_loss_fake\n",
        "\n",
        "            g_loss = tf.reduce_mean(g_loss)\n",
        "            d_loss = tf.reduce_mean(d_loss)\n",
        "\n",
        "        gradients_of_generator = tape_g.gradient(g_loss, generator.trainable_variables)\n",
        "        gradients_of_discriminator = tape_d.gradient(d_loss, discriminator.trainable_variables)\n",
        "\n",
        "        optimizer_g.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))\n",
        "        optimizer_d.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))\n",
        "\n",
        "        return g_loss, d_loss\n",
        "\n",
        "\n",
        "\n",
        "    for epoch in range(epochs):\n",
        "        for batch in range(0, len(data), batch_size):\n",
        "            end_index = batch + batch_size\n",
        "            if end_index > len(data):\n",
        "                end_index = len(data)\n",
        "            real_data_batch = data[batch:end_index]\n",
        "            g_loss, d_loss = train_step(real_data_batch)\n",
        "\n",
        "        print(f'Epoch {epoch + 1}, Generator Loss: {g_loss}, Discriminator Loss: {d_loss}')\n",
        "\n",
        "        total_loss = g_loss + d_loss\n",
        "        if best_loss - total_loss > min_delta:\n",
        "           best_loss = total_loss\n",
        "           epochs_no_improve = 0\n",
        "        else:\n",
        "            epochs_no_improve += 1\n",
        "\n",
        "\n",
        "        if epochs_no_improve >= patience:\n",
        "            print(f'Early stopping has occured at epoch {epoch}')\n",
        "            break\n",
        "\n",
        "    generator.save('stock_generator.tf')\n",
        "    discriminator.save('stock_discriminator.tf')"
      ],
      "metadata": {
        "id": "9s5ypig6ADrs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "train_size = int(df.shape[0] * 0.8)\n",
        "test_size = df.shape[0] - train_size\n",
        "train_df = df.iloc[:train_size]\n",
        "# test_data = df.iloc[train_size:]\n",
        "train_data = create_data_vector(train_df, window_size)\n",
        "# print(train_data.shape)\n",
        "# 398 3\n",
        "input_shape = (train_data.shape[1], train_data.shape[2])\n",
        "generator = build_generator(input_shape)\n",
        "discriminator = build_discriminator(input_shape)\n",
        "\n",
        "train_gan(generator, discriminator, train_data)"
      ],
      "metadata": {
        "id": "yOHz9Au9ALRG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "def model_evaluate(generator, test_size, df, window_size):\n",
        "\n",
        "    results = {}\n",
        "\n",
        "    for x in range(nist):\n",
        "        results[f'stock_{x+1}'] = {\n",
        "            'Day': [],\n",
        "            'Predicted Price': [],\n",
        "            'Actual Price': [],\n",
        "            'Absolute Difference': [],\n",
        "            'Direction Match': []\n",
        "        }\n",
        "\n",
        "\n",
        "    for stock in range(1):\n",
        "\n",
        "        for i in range(test_size):\n",
        "            start = df.shape[0] - test_size + i - window_size\n",
        "            end = df.shape[0] - test_size + i\n",
        "\n",
        "            print(f'{start} - {end}')\n",
        "\n",
        "            test_data_batch = df.iloc[start:end, stock]\n",
        "\n",
        "            # print(test_data_batch.shape)\n",
        "\n",
        "            normalized_data, norms, scaler, fft_list = nomralize_data(test_data_batch.values.flatten(), m, tau)\n",
        "            # normalized_stock_df = pd.DataFrame(normalized_stock_data, columns=[f'normalized_{j+1}' for j in range(m)])\n",
        "            normalized_data = normalized_data.reshape(1, normalized_data.shape[0], normalized_data.shape[1])\n",
        "\n",
        "            # print(normalized_stock_data.shape)\n",
        "            # normalized_stock_df = normalized_stock_df.values.reshape(1, window_size, 3)\n",
        "\n",
        "            predicted = generator.predict(normalized_data)\n",
        "            predicted_price = denormalize_data(predicted, norms[-1:], scaler, fft_list).flatten()[-1]\n",
        "\n",
        "\n",
        "            actual_price = test_data_batch.values.flatten()[-1]\n",
        "            # actual_prices = test_data_batch.values.flatten()\n",
        "            # predicted_prices = denormalize_data(predicted, norms[-1:], scaler, fft_list).flatten()\n",
        "            # print(predicted_prices)\n",
        "\n",
        "            # print(actual_prices)\n",
        "            # time_index = np.arange(len(actual_prices))\n",
        "\n",
        "            # plt.figure(figsize=(10, 5))\n",
        "            # plt.plot(time_index, actual_prices, label='Actual Prices', color='blue')\n",
        "            # plt.plot(time_index, predicted_prices, label='Predicted Prices', color='red')\n",
        "            # plt.title('Comparison of Actual and Predicted Prices')\n",
        "            # plt.xlabel('Time Index')\n",
        "            # plt.ylabel('Price')\n",
        "            # plt.legend()\n",
        "            # plt.grid(True)\n",
        "            # plt.show()\n",
        "\n",
        "            stock_key = f'stock_{stock + 1}'\n",
        "\n",
        "\n",
        "\n",
        "            print(f'{start} to {end} - actual price: {actual_price} vs predicted price: {predicted_price}')\n",
        "            difference = np.abs(predicted_price - actual_price)\n",
        "            direction_match = False\n",
        "\n",
        "            prev_price =  df.iloc[start - 1, stock]\n",
        "            actual_change = (actual_price - prev_price) / prev_price\n",
        "            predicted_change = (predicted_price - prev_price) / prev_price\n",
        "            if (actual_change > 0 and predicted_change > 0) or (actual_change < 0 and predicted_change < 0):\n",
        "                direction_match = True\n",
        "\n",
        "\n",
        "\n",
        "            results[stock_key]['Day'].append(i + window_size)\n",
        "            results[stock_key]['Predicted Price'].append(predicted_price)\n",
        "            results[stock_key]['Actual Price'].append(actual_price)\n",
        "            results[stock_key]['Absolute Difference'].append(difference)\n",
        "            results[stock_key]['Direction Match'].append(direction_match)\n",
        "\n",
        "    results_df = pd.DataFrame(results)\n",
        "    results_df.to_csv('test_results.csv', index=False)\n",
        "    print(\"Test results saved to 'test_results.csv'.\")\n",
        "\n",
        "generator = load_model('stock_generator.keras')\n",
        "model_evaluate(generator, test_size, df, window_size)\n",
        "# print(window_size)\n",
        "\n",
        "# print(df.shape)\n",
        "\n"
      ],
      "metadata": {
        "id": "Uk7hCzQtAOn0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "results_df = pd.read_csv('test_results.csv')\n",
        "\n",
        "number = 1\n",
        "stock_no_results = results_df[f'stock_{number}'].apply(eval).apply(pd.Series)\n",
        "\n",
        "predicted_prices = stock_no_results.iloc[1]\n",
        "actual_prices = stock_no_results.iloc[2]\n",
        "days = range(len(predicted_prices))\n",
        "\n",
        "plt.figure(figsize=(12, 6))\n",
        "plt.plot(days, actual_prices, label='Actual Prices', color='blue', linestyle='-')\n",
        "plt.plot(days, predicted_prices, label='Predicted Prices', color='red', linestyle='--')\n",
        "plt.title('Comparison of Actual and Predicted Prices')\n",
        "plt.xlabel('Evaluation Instance')\n",
        "plt.ylabel('Price')\n",
        "plt.legend()\n",
        "plt.grid(True)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "NuwxooO93kb5"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}