Skip to content

[SPARK-24573][INFRA] Runs SBT checkstyle after the build to work around a side-effect #21579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 10 commits into from
Closed

[SPARK-24573][INFRA] Runs SBT checkstyle after the build to work around a side-effect #21579

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
8307cbd
Empty commit
HyukjinKwon Jun 17, 2018
96f5458
Java change
HyukjinKwon Jun 17, 2018
01a9bc8
Spectulcation
HyukjinKwon Jun 17, 2018
7f46304
Does it still check the style correctly
HyukjinKwon Jun 17, 2018
ed50f6d
Revert the java change back
HyukjinKwon Jun 17, 2018
14351d6
Unidoc check
HyukjinKwon Jun 17, 2018
6529622
Revert "Unidoc check"
HyukjinKwon Jun 17, 2018
73a494e
Fix comment
HyukjinKwon Jun 17, 2018
7eba44a
Java change
HyukjinKwon Jun 17, 2018
d902b10
Revert "Java change"
HyukjinKwon Jun 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,7 @@ def build_spark_unidoc_sbt(hadoop_version):
exec_sbt(profiles_and_goals)


def build_spark_assembly_sbt(hadoop_version):
def build_spark_assembly_sbt(hadoop_version, checkstyle=False):
# Enable all of the profiles for the build:
build_profiles = get_hadoop_profiles(hadoop_version) + modules.root.build_profile_flags
sbt_goals = ["assembly/package"]
Expand All @@ -366,6 +366,9 @@ def build_spark_assembly_sbt(hadoop_version):
" ".join(profiles_and_goals))
exec_sbt(profiles_and_goals)

if checkstyle:
run_java_style_checks()
Copy link
Member Author

@HyukjinKwon HyukjinKwon Jun 17, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This takes 2-ish mins now given my past tries.


# Note that we skip Unidoc build only if Hadoop 2.6 is explicitly set in this SBT build.
# Due to a different dependency resolution in SBT & Unidoc by an unknown reason, the
# documentation build fails on a specific machine & environment in Jenkins but it was unable
Expand Down Expand Up @@ -570,11 +573,13 @@ def main():
or f.endswith("scalastyle-config.xml")
for f in changed_files):
run_scala_style_checks()
should_run_java_style_checks = False
if not changed_files or any(f.endswith(".java")
or f.endswith("checkstyle.xml")
or f.endswith("checkstyle-suppressions.xml")
for f in changed_files):
run_java_style_checks()
# Run SBT Checkstyle after the build to prevent a side-effect to the build.
should_run_java_style_checks = True
if not changed_files or any(f.endswith("lint-python")
or f.endswith("tox.ini")
or f.endswith(".py")
Expand Down Expand Up @@ -603,7 +608,7 @@ def main():
detect_binary_inop_with_mima(hadoop_version)
# Since we did not build assembly/package before running dev/mima, we need to
# do it here because the tests still rely on it; see SPARK-13294 for details.
build_spark_assembly_sbt(hadoop_version)
build_spark_assembly_sbt(hadoop_version, should_run_java_style_checks)

# run the test suites
run_scala_tests(build_tool, hadoop_version, test_modules, excluded_tags)
Expand Down