Skip to content

[SPARK-4137] [EC2] Don't change working dir on user #2988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-4137] [EC2] Don't change working dir on user #2988

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ce071fc
don't change working dir
nchammas Oct 29, 2014
77871a2
add clarifying comment
nchammas Oct 29, 2014
bcdf6a5
fix typo
nchammas Oct 29, 2014
752f958
specify deploy.generic path absolutely
nchammas Oct 30, 2014
fbc20c7
revert to old commenting style
nchammas Nov 6, 2014
f3850b5
pep8 fix
nchammas Nov 6, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions ec2/spark-ec2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,9 @@
# limitations under the License.
#

cd "`dirname $0`"
PYTHONPATH="./third_party/boto-2.4.1.zip/boto-2.4.1:$PYTHONPATH" python ./spark_ec2.py "$@"
# Preserve the user's CWD so that relative paths are passed correctly to
#+ the underlying Python script.
SPARK_EC2_DIR="$(dirname $0)"

PYTHONPATH="${SPARK_EC2_DIR}/third_party/boto-2.4.1.zip/boto-2.4.1:$PYTHONPATH" \
python "${SPARK_EC2_DIR}/spark_ec2.py" "$@"
12 changes: 11 additions & 1 deletion ec2/spark_ec2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
from boto import ec2

DEFAULT_SPARK_VERSION = "1.1.0"
SPARK_EC2_DIR = os.path.dirname(os.path.realpath(__file__))

# A URL prefix from which to fetch AMI information
AMI_PREFIX = "https://raw.github.com/mesos/spark-ec2/v2/ami-list"
Expand Down Expand Up @@ -586,7 +587,14 @@ def setup_cluster(conn, master_nodes, slave_nodes, opts, deploy_ssh_key):
ssh(master, opts, "rm -rf spark-ec2 && git clone https://github.com/mesos/spark-ec2.git -b v4")

print "Deploying files to master..."
deploy_files(conn, "deploy.generic", opts, master_nodes, slave_nodes, modules)
deploy_files(
conn=conn,
root_dir=SPARK_EC2_DIR + "/" + "deploy.generic",
opts=opts,
master_nodes=master_nodes,
slave_nodes=slave_nodes,
modules=modules
)

print "Running setup on master..."
setup_spark_cluster(master, opts)
Expand Down Expand Up @@ -723,6 +731,8 @@ def get_num_disks(instance_type):
# cluster (e.g. lists of masters and slaves). Files are only deployed to
# the first master instance in the cluster, and we expect the setup
# script to be run on that instance to copy them to other nodes.
#
# root_dir should be an absolute path to the directory with the files we want to deploy.
def deploy_files(conn, root_dir, opts, master_nodes, slave_nodes, modules):
active_master = master_nodes[0].public_dns_name

Expand Down