This code performs analysis on a Walmart stock dataset using Spark. It utilizes various Spark functions to explore and manipulate the data. Here is a breakdown of the tasks performed:
- The code reads a CSV file containing Walmart stock data into a DataFrame.
- Printing the number of rows in the DataFrame.
- Printing the column names.
- Printing the schema of the DataFrame.
- Printing the first 5 lines of the DataFrame.
- Using the describe() function to generate summary statistics of the DataFrame.
- Formatting the decimal places of the summary statistics to two decimal places.
- Converting certain columns to decimal type.
- Creating a new column called "HV Ratio" that represents the ratio of the High Price to the volume of stock traded for a day.
- Computing the mean of the "Close" column.
- Finding the maximum and minimum values of the "Volume" column.
- Counting the number of days where the "Close" price was lower than $60.
- Calculating the percentage of time the "High" price was greater than $80.
- Finding the maximum "High" price for each year.
- Calculating the average "Close" price for each calendar month.
- This code can be used as a starting point for analyzing and gaining insights from the Walmart stock dataset using Spark.
- You can get the datasets from the kaggle :- https://www.kaggle.com/