-
Notifications
You must be signed in to change notification settings - Fork 1
/
readme.txt
153 lines (109 loc) · 7.53 KB
/
readme.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
============ Code Explanation ============
Shared Memory:
This python code is a parallel external sort. It is implemented with multi-processing. First, it reads the unsorted file into chunks by block size (memory size). Then it sorts the spitted files parallelly and merge all the chunks. Finally, it generates the output and write it into a single file.
MapReduce:
This program is designed to read the input files with keys as the first ten bytes of data and rest of the line as value. Mappers and reducers are defined for every setup and each mapper processes a part of input file and writes the intermediate file to be picked up by reducer phase. In the reducer phase the intermediate files are shuffled and sorted and the reducers perform reduce actions and write it to the specified hdfs folder.
Spark:
Input files are stored as RDDs. Transformations are applied on these RDDs to perform the split to generate key value pairs of input file. These transformed RDDs are then processed by sorting each record by key value to generate the sorted version of the input data. Spark performs operations only when actions are specified and not for the transformations. Output generated is then stored to the external hdfs file for validation.
============ Usage ============
How to start Shared-Memory TeraSort:
First, change the current directory to the folder “PA2_Liu_Pereira”:
Copy the source code python file and the auto-testing shell script.
`cp src/shared_memory_sort.py ~/PA2_Liu_Pereira/`
`cp Scripts/run_shared_menory.sh ~/PA2_Liu_Pereira/`
Then Run the script to start auto testing (Change the data size for testing in the script if needed
e.g. `./gensort -a 10000000000 testfile` for 1TB, 1280000000 for 128GB ):
`./run_shared_menory.sh`
> You will need "gensort" and "valsort" from [Sortbenchmark](http://sortbenchmark.org)
For different configurations:
The file shared_memory_sort.py can be provided by the following parameters:
python shared_memory_sort.py number_of_proccess input_file_name output_file_name -b block_szie -k key -t temp_path
e.g. `python shared_memory_sort.py 3 input output -b 55M -k "line[0:10]" -t /mnt`
| Args | Value |
|:------:| ---------------------- |
| 0 | number of threads |
| 1 | input file |
| 2 | output file |
| -b | block size e.g. 50M 16G|
| -k | key |
| -t | temporary file path |
If any permission issues show up, please do `chmod` for the permission:
`chmod +x run_shared_menory.sh`
If it shows "Killed", please use `free` to check your memory usage. Do not put the block size with your full memory capacity without checking.
After using the sorting, please make sure all the processes are killed.
You may do it by using the following command:
`ps -fA | grep python`
> `kill PID` # if they is remaining
How to do Disks mounting:
You may want to put the entire directory under mounted disk ( e.g. /mnt ) for testing large file.
Simply, run the following scripts:
One disk:
`./Scripts/one_node_mount.sh`
Multiple disks:
`./Scripts/mount.sh`
For the following, if you wamt to use it with i3.4xlarge, please use those files ending with "_4xlarge".
Or if you wamt to use it with 8nodes, please use those files ending with "_8nodes.
How to install Hadoop?
1. Change the directory to point hadoop_config.sh under PA2_Liu_Pereira/Scripts directory.
2. Run the script as follows., `./ hadoop_config.sh`
3. Verify .bashrc file to ensure required hadoop paths are added.
4. Change the working directory to hadoop config directory for. Eg., /mnt/hadoop/etc/hadoop/
5. Modify core-site.xml, mapred-site.xml, hdfs-site.xml, yarn-site.xml as per the files under PA2_Liu_Pereira/Config for 1-node and 8-nodes cluster respectively.
6. Once the required configurations are made, you can start running hadoop services using start-all.sh (start-dfs.sh and start-yarn.sh).
7. Check using ‘jps’ command to ensure hadoop is up and running.
8. Move the input file generated by gensort to hdfs using the commands below.
`hdfs dfs -mkdir /input`
`hdfs dfs -put pinput /input`
9. For 8 nodes, along with this add slaves,masters file from
PA2_Liu_Pereira/Config folder to /mnt/hadoop/etc/hadoop/
10. To setup slaves, make a snapshot of master (as followed from step 1-9) and install in 8 slaves.
Remember to exclude masters file from /mnt/hadoop/etc/hadoop/ in slave VMs.
11. For 8 nodes, add this config in master node.
a. Add identity file info, host name,username in ~/.ssh/config file which is under PA2_Liu_Pereira/MultiNode/config
b. Add all VM ips in the /etc/hosts file which is under PA2_Liu_Pereira/MultiNode/config
How to install Spark on Hadoop?
1. Change the directory to point spark_config.sh under PA2_Liu_Pereira/Scripts directory.
2. Run the script as follows., ./ spark_config.sh
3. Verify .bashrc file to ensure required spark paths are added.
4. Ensure access keys are added accordingly.
5. Change the working directory to spark directory for. Eg., /mnt/spark/
3. To start the spark cluster follow below steps:
a. Change directory to point to ec2 folder using cd /mnt/spark/ec2
b. Run the following command to launch cluster:
`./spark-ec2 -k i3keypair -i ~/i3keypair.pem --instance-type=i3.large launch Spark-1node`
`./spark-ec2 -k aws -i ~/aws.pem --instance-type=i3.4xlarge launch Spark-1node`
`./spark-ec2 -k i3keypair -i ~/i3keypair.pem --instance-type=i3.large --slaves=8 launch SparkCluster`
7. Connection will be refused multiple times, if you get disconnected run using resume parameter as follows. In some cases, it might take upto 10 minutes in some cases to launch the cluster.
`./spark-ec2 -k i3keypair -i ~/i3keypair.pem --instance-type=i3.large launch Spark-1node --resume`
Running MapReduce Program:
Once the hadoop services are running , we can run the hadoop mapreduce program by following
below steps.
Step 1: Create the jar of the map reduce program (HadoopSort.java) by changing directory to PA2_Liu_Pereira/src.
`hadoop com.sun.tools.javac.Main HadoopSort.java`
`jar cf sort.jar HadoopSort*.class`
Step 2: Run the program using below command
`hadoop jar sort.jar HadoopSort /input/pinput /poutput`
Step 3: Validate the results in hdfs under /poutput by viewing hdfs dfs -ls /poutput
Running Spark Program:
1. Once the cluster has been launched you can start running spark program by submitting the task to spark master node using below command.
`/spark-submit /mnt/src/sparksort.py /input/pinput /sparkout`
where arg1 is spark command, arg2 is spark program, arg3 is input file path in hdfs, arg4 is output path to store results in hdfs.
2. Ensure when running the multi node spark program for 8 nodes, all the slaves are up and running while running spark program
3. Validate the results generated in hdfs by using hdfs dfs -ls /sparkout
Validating the output is sorted?
Run the valsort program and pass the output file generated for checksum and descriptions.
`./valsort sortedFile`
Tips to troubleshoot:
● If the namenode is not starting with other nodes you can follow below steps to get it working.
`stop-all.sh`
`hadoop namenode -format`
`start-all.sh`
● If it still doesn't work, try below steps.
`rm -r namenode rm -r datanode`
In */hadoop_store/hdfs directory use
`sudo mkdir namenode`
`sudo mkdir datanode`
`chmod -R 755 namenode and chmod -R 755 datanode`
Restart services using start-all.sh
● In case of user permission issue while starting namenode in hadoop services using start-all.sh,
`sudo chown -R ubuntu /mnt/[hadoop_directory]/`