Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-25875][k8s] Merge code to set up driver command into a single step. #22897

Closed
wants to merge 9 commits into from

Commits on Oct 30, 2018

  1. [SPARK-25875][k8s] Merge code to set up driver command into a single …

    …step.
    
    Right now there are 3 different classes dealing with building the driver
    command to run inside the pod, one for each "binding" supported by Spark.
    This has two main shortcomings:
    
    - the code in the 3 classes is very similar; changing things in one place
      would probably mean making a similar change in the others.
    
    - it gives the false impression that the step implementation is the only
      place where binding-specific logic is needed. That is not true; there
      was code in KubernetesConf that was binding-specific, and there's also
      code in the executor-specific config step. So the 3 classes weren't really
      working as a language-specific abstraction.
    
    On top of that, the current code was propagating command line parameters in
    a different way depending on the binding. That doesn't seem necessary, and
    in fact using environment variables for command line parameters is in general
    a really bad idea, since you can't handle special characters (e.g. spaces)
    that way.
    
    This change merges the 3 different code paths for Java, Python and R into
    a single step, and also merges the 3 code paths to start the Spark driver
    in the k8s entry point script. This increases the amount of shared code,
    and also moves more feature logic into the step itself, so it doesn't live
    in KubernetesConf.
    
    Note that not all logic related to setting up the driver lives in that
    step. For example, the memory overhead calculation still lives separately,
    except it now happens in the driver config step instead of outside the
    step hierarchy altogether.
    
    Some of the noise in the diff is because of changes to KubernetesConf, which
    will be addressed in a separate change.
    
    Tested with new and updated unit tests + integration tests.
    Marcelo Vanzin committed Oct 30, 2018
    Configuration menu
    Copy the full SHA
    8148f49 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2018

  1. Merge branch 'master' into SPARK-25875

    Marcelo Vanzin committed Oct 31, 2018
    Configuration menu
    Copy the full SHA
    b118b91 View commit details
    Browse the repository at this point in the history
  2. Tweaks after merge.

    Marcelo Vanzin committed Oct 31, 2018
    Configuration menu
    Copy the full SHA
    91a4741 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2018

  1. match -> if

    Marcelo Vanzin committed Nov 1, 2018
    Configuration menu
    Copy the full SHA
    1a61f72 View commit details
    Browse the repository at this point in the history
  2. Make mainAppResource not optional.

    Marcelo Vanzin committed Nov 1, 2018
    Configuration menu
    Copy the full SHA
    91765ab View commit details
    Browse the repository at this point in the history
  3. Remove obsolete test.

    Marcelo Vanzin committed Nov 1, 2018
    Configuration menu
    Copy the full SHA
    5601d16 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2018

  1. Temporary debug change to see what's failing.

    Marcelo Vanzin committed Nov 2, 2018
    Configuration menu
    Copy the full SHA
    4152c76 View commit details
    Browse the repository at this point in the history
  2. Merge branch 'master' into SPARK-25875

    Marcelo Vanzin committed Nov 2, 2018
    Configuration menu
    Copy the full SHA
    da4856e View commit details
    Browse the repository at this point in the history
  3. Add constants for resource types.

    Marcelo Vanzin committed Nov 2, 2018
    Configuration menu
    Copy the full SHA
    0d755d6 View commit details
    Browse the repository at this point in the history