Automated creating and updating scala_x_x.bzl repository files #1608
Conversation
lm1nrt
force-pushed
the
michal-lenart/automated-version-update
branch
from
c703f8e
to
3248e01
Compare
| scala_major = ".".join(scala_version.split(".")[:2]) | ||
| scala_test_major = "3" if scala_major >= "3.0" else scala_major | ||
| scala_fmt_major = "2.13" if scala_major >= "3.0" else scala_major | ||
| kind_projector_version = "0.13.2" if scala_major < "2.13" else "0.13.3" |
There was a problem hiding this comment.
Some versions are not compatible/doesn't exist in certain Scala version
|
|
||
| def get_maven_coordinates(artifact) -> MavenCoordinates: | ||
| splitted = artifact.split(':') | ||
| version = splitted[2] if splitted[2][0].isnumeric() else splitted[3] |
There was a problem hiding this comment.
The version location in maven coordinates isn't strict, so there are cases that the version is under the x+1 index after splitting it by ':'
| data = json.load(file) | ||
| return get_mavens_coordinates_from_json(dependency["directDependencies"]) if any((dependency := d)["coord"] == artifact for d in data["dependencies"]) else [] | ||
|
|
||
| def get_label(coordinate) -> str: |
There was a problem hiding this comment.
There are 5 patterns of labels, which are related to the structure of certain group and artifact due to custom naming.
| temp = {} | ||
|
|
||
| for a in artifacts: | ||
| label = get_label(a.coordinates).replace('scala3_', 'scala_').replace('scala_tasty_core', 'scala_scala_tasty_core') |
There was a problem hiding this comment.
It's related to custom naming convention of certain labels, TBD in the future to simplify it.
| def is_that_trailing_coma(content, char, indice) -> bool: | ||
| return content[indice] == char and content[indice+1] != ',' and content[indice+1] != ':' and content[indice+1] != '@' and not content[indice+1].isalnum() | ||
|
|
||
| def get_with_trailing_commas(content) -> str: |
There was a problem hiding this comment.
After mapping to the dict, there are no trailing commas, so this is a workaround to adjust file to .bzl files / starlark convention and CI-checks failures.
| path = os.getcwd().replace('/scripts', '/third_party/repositories') | ||
| file = Path(f'{path}/{'scala_' + "_".join(version.split(".")[:2]) + '.bzl'}') | ||
|
|
||
| if not file.exists(): |
There was a problem hiding this comment.
If file doesn't exist, the script is copying the content from last file in directory (default sort) and updating it. So it won't work when someone is creating e.g. scala_2_10.bzl (since the heuristic is copying the content from scala_3_5.bzl in this example).
| data.write(line) | ||
|
|
||
| with file.open('r+') as data: | ||
| excluded_artifacts = ["org.scala-lang.modules:scala-parser-combinators_2.11:1.0.4"] |
There was a problem hiding this comment.
There are artifacts versions that cause CI-check failures after update and exceptions in logs.
| original_artifact_dict[label]["artifact"] = artifact | ||
| original_artifact_dict[label]["sha256"] = sha | ||
| if deps: | ||
| dependencies = [d for d in deps if d[1:] in labels and "runtime" not in d and "runtime" not in artifact] |
There was a problem hiding this comment.
Not modifying the runtime dependencies due to CI-checks failure.
lm1nrt
changed the title
lm1nrt
changed the title
lm1nrt
changed the title
|
|
||
| def map_to_resolved_artifacts(output) -> List[ResolvedArtifact]: | ||
| resolved_artifacts = [] | ||
| subprocess.call(f'cs fetch {' '.join(output)} --json-output-file out.json', shell=True) |
There was a problem hiding this comment.
Creates local file to get the directDependencies for fetched artifacts
| @@ -2,68 +2,144 @@ scala_version = "2.11.12" | |||
|
|
|||
There was a problem hiding this comment.
Right now scala_version variable is needed due to repositories.bzl file and other usages, so it can be changed in the future.
|
@liucijus @simuons Could you take a look at this PR? It is a first step to improve maven dependency management in rules_scala. It will allow to bump patch versions of scala and add new minor versions faster. Eventually we would like to organize dependencies more, to keep naming consistent, have clear understanding which deps belong into which resolution tree, etc. and have more frequent updates of other dependencies too. |
Description
It's not the ideal version, but I believe it can be improved by adding support for more Scala artifacts.
Motivation
Updating and creating new versions for scala_x_x.bzl files is time-consuming, and it's a kind of work that is not performed regularly due to its specifics.