chore: transfer java-related synthtool code into the library_generation image #3011

diegomarquezp · 2024-07-03T20:11:03Z

This PR moves java.py and the minimum required dependencies from synthtool.

It contains the changes of @JoeWang1127's googleapis/synthtool@696c4bf

Details of code transfer

Some files have been simplified and others simply verbatim-copied. Here is a detailed list of changes (the paths are relative to library_generation/owlbot:

synthtool/__init__.py: No changes made. This is the module initialization configuration to allow statements such as import synthtool as s combined with s.copy, s.replace, etc. Without this, we would have to use import synthtool.transforms as s.
synthtool/_tracked_paths.py: No changes made. It allows to use relative paths internally when working with library files.
synthtool/gcp/common.py: Simplified to use java-only functions. This file originally had a set of functions to support postprocessing of languages written in multiple languages. The most important function is common_templates, which renders the templates for the library (e.g. workflow files, kokoro files). Note that common_templates was modified in order to require and only allow specifying the path to the templates via the SYNTHTOOL_TEMPLATES env var, as opposed to its original support of 3 separate ways, including cloning synthtool and reading the templates from there, because this is now an internal detail of how the library_generation image will work.
synthtool/gcp/samples.py: No changes made. This is a helper to obtain path and metadata about the generated samples of a library. It is then used when rendering the README.
synthtool/gcp/snippets.py: No changes made. Similar to samples.py in the way it's used to render a library's README
synthool/languages/java.py: Small modifications around the fact that we dropped several files (e.g. use gcp.common.CommonTemplates instead of gcp.CommonTemplates to save us from an extra gcp/__init__.py).
synthtool/sources/templates.py: No changes made. Internally used by common_templates. Contains the underlying usage of jinja2 to render the templates.
synthtool/transforms.py: No changes made. Contains a few functions that are commonly used by owlbot.py files (example)

Changes in Dockerfile

We will not clone synthtool anymore. We will instead install it as a separate package whose source code is within sdk-platform-java.

snippet-bot · 2024-07-03T21:30:38Z

Here is the summary of possible violations 😱

There is a possible violation for not having product prefix.

library_generation/owlbot/synthtool/gcp/snippets.py:20, tag ([a-z0-9_

There is a format violation for a region tag.

library_generation/owlbot/synthtool/gcp/snippets.py:20, tag ([a-z0-9_

The end of the violation section. All the stuff below is FYI purposes only.

Here is the summary of changes.

You are about to add 1 region tag.

library_generation/owlbot/synthtool/gcp/snippets.py:20, tag ([a-z0-9_

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

JoeWang1127 · 2024-07-04T13:25:15Z

.cloudbuild/library_generation/library_generation.Dockerfile

-RUN python3 -m pip install -r requirements.in
+# install java-only synthtool scripts as a python package
+WORKDIR /src/owlbot/synthtool
+RUN python -m pip install -r requirements.txt


Can we combine this requirements.txt with library_generation/requirements.txt?

I renamed library_generation/requirementss.txt to requirementss.in (base dependencies) and created a hashed requirements.txt. This is in addition of combining the dependency files of owlbot/synthool

JoeWang1127 · 2024-07-04T13:28:03Z

library_generation/owlbot/synthtool/gcp/common.py

+
+    # This default should be switched to "main" once we've migrated
+    # the majority of our repositories:
+    return os.getenv("DEFAULT_BRANCH", "master")


Should we change the default value to main?

True, but I just realized this is used to compute a template value not used in our templates

JoeWang1127 · 2024-07-04T13:30:42Z

library_generation/owlbot/synthtool/gcp/common.py

+        t = templates.TemplateGroup(self._template_root / directory, self.excludes)
+
+        if "repository" in kwargs["metadata"] and "repo" in kwargs["metadata"]:
+            kwargs["metadata"]["repo"]["default_branch"] = _get_default_branch_name(


Do you know which template uses this key?

No template uses it, removing.
PS: I saw this comment after reading #3011 (comment)

JoeWang1127 · 2024-07-04T13:33:29Z

library_generation/owlbot/synthtool/gcp/samples.py

Do you know how this file and its functions are used?

From the PR description, the functions are used to render the README in java.py

sdk-platform-java/library_generation/owlbot/synthtool/languages/java.py

Line 288 in 635d92a

metadata["samples"] = samples.all_samples(["samples/**/src/main/java/**/*.java"])

There is a link to the part of the README template where this is used.

JoeWang1127 · 2024-07-04T13:34:01Z

library_generation/owlbot/synthtool/gcp/snippets.py

Same comment here, do you know how this file and its functions are used?

From the PR description, the functions are used to render the README in java.py

sdk-platform-java/library_generation/owlbot/synthtool/languages/java.py

Lines 289 to 291 in 635d92a

metadata["snippets"] = snippets.all_snippets(

["samples/**/src/main/java/**/*.java", "samples/**/pom.xml"]

)

JoeWang1127 · 2024-07-04T13:36:26Z

library_generation/owlbot/synthtool/languages/java.py

+    )
+
+    # skip README generation on Kokoro (autosynth)
+    if os.environ.get("KOKORO_ROOT") is not None:


I think we can remove this line since we won't run tests/build in Kokoro.

Good point. Removed

JoeWang1127 · 2024-07-04T13:37:23Z

library_generation/owlbot/synthtool/languages/java.py

+    )
+
+
+def latest_maven_version(group_id: str, artifact_id: str) -> Optional[str]:


I think we don't need this function once #3007 is merged. We can keep it for now.

Now #3007 is merged, could you create a follow up PR to remove this function?

JoeWang1127 · 2024-07-04T13:38:57Z

library_generation/owlbot/synthtool/languages/java.py

+    cloud_prefix = "cloud-" if cloud_api else ""
+    package_name = package_pattern.format(service=service, version=version)
+    fix_proto_headers(
+        library / f"proto-google-{cloud_prefix}{service}-{version}{suffix}"


nit: could we use a variable for f"proto-google-{cloud_prefix}{service}-{version}{suffix}"?

JoeWang1127 · 2024-07-04T13:45:40Z

library_generation/owlbot/synthtool/transforms.py

Could you search owlbot.py in google-cloud-java and handwritten libraries to see whether there are unused functions and remove them from this file?

Thanks, there was one function dont_overwrite that wasn't used.

JoeWang1127 · 2024-07-04T19:06:10Z

library_generation/requirements.in

+nox==2024.4.15
+requests==2.32.3
+setuptools==65.5.1
+jinja2==3.1.4


this package appears twice.

JoeWang1127 · 2024-07-04T19:07:51Z

library_generation/requirements.in

+setuptools==65.5.1
+jinja2==3.1.4
+deprecation==2.1.0
+protobuf==3.20.2


Did you find which functions use deprecation and protobuf package?

I didn't look at the functions but tried running the generation without these. It worked, so I'm removing them. Thanks for catching this.

JoeWang1127 · 2024-07-04T19:09:16Z

Could you fix the error in unit tests and lint?

diegomarquezp · 2024-07-05T16:14:08Z

Could you fix the error in unit tests and lint?

I had to pin a transitive dependency to a specific version (added a comment in requirements.in)

JoeWang1127 · 2024-07-05T18:43:43Z

I had to pin a transitive dependency to a specific version (added a comment in requirements.in)

Could you enrich the comment about the error and why pinning this dependency can fix the issue.

sonarcloud · 2024-07-08T13:06:41Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarcloud · 2024-07-08T13:10:04Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

diegomarquezp added 12 commits July 3, 2024 16:56

chore: move relevant synthtool source into hermetic docker image

a3e1164

restructure scripts

75684bd

simplify gcp.common

30fa2c4

simplify java.py

af71764

add transforms

c61a80c

fix runtime errors

13a27a9

exclude usage of metadata

46c841c

fix logger creation

f2fb117

remove unused cache and shell imports

420d7f4

bring partials.py into gcp/common.py

09af9a9

use transferred partials in java.py

b1d33e3

final adjustments, add samples and snippets

fb3e399

product-auto-label bot added the size: xl Pull request size is extra large. label Jul 3, 2024

diegomarquezp added 5 commits July 3, 2024 20:24

add requirements files

e4e1770

install java-only synthool in dockerfile

d93093d

remove unused languages/common.py

80e008d

add changes of googleapis/synthtool@696c4bf

5fd8f9e

lint

635d92a

diegomarquezp marked this pull request as ready for review July 3, 2024 21:30

diegomarquezp requested a review from a team as a code owner July 3, 2024 21:30

diegomarquezp requested a review from JoeWang1127 July 3, 2024 21:30

JoeWang1127 reviewed Jul 4, 2024

View reviewed changes

diegomarquezp added 6 commits July 4, 2024 15:28

remove unused transforms function

8dbdd90

merge setup.py files

b25bd60

remove unused default_branch function

643f957

merge requirements file

f03aac3

use base requirements file

3c6d02a

remove installation of synthool in favor of combined installation

5d99c69

diegomarquezp requested a review from JoeWang1127 July 4, 2024 15:58

diegomarquezp added 3 commits July 4, 2024 16:02

remove unused function

fd39dad

use variable names for java.py

35e27da

use variables for key references in java.py

605266d

diegomarquezp mentioned this pull request Jul 4, 2024

feat: get released version from versions.txt to render README.md #3007

Merged

JoeWang1127 reviewed Jul 4, 2024

View reviewed changes

diegomarquezp added 6 commits July 5, 2024 10:51

Merge branch 'main' into transfer-synthtool

9530478

remove duplicated dependency

c7f86e2

lint

6698065

restore format of golden owlbot.py

065273e

reduce unnecessary dependencies

6186e79

pin typing extensions

6ab904b

JoeWang1127 approved these changes Jul 5, 2024

View reviewed changes

diegomarquezp added 2 commits July 5, 2024 19:50

enrich comment of the typing-extensions dependency in requirements.in

0780ad1

make snippet bot to ignore the synthtool snippet script

7c56ce0

diegomarquezp merged commit 7ccec6d into main Jul 8, 2024
47 of 51 checks passed

diegomarquezp deleted the transfer-synthtool branch July 8, 2024 13:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: transfer java-related synthtool code into the library_generation image #3011

chore: transfer java-related synthtool code into the library_generation image #3011

diegomarquezp commented Jul 3, 2024 •

edited

Loading

snippet-bot bot commented Jul 3, 2024 •

edited

Loading

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 4, 2024

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 4, 2024

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 4, 2024 •

edited

Loading

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 4, 2024

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 4, 2024

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 4, 2024

JoeWang1127 Jul 4, 2024

JoeWang1127 Jul 5, 2024

JoeWang1127 Jul 4, 2024 •

edited

Loading

diegomarquezp Jul 4, 2024

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 4, 2024

diegomarquezp Jul 4, 2024

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 5, 2024

JoeWang1127 Jul 4, 2024

diegomarquezp Jul 5, 2024

JoeWang1127 commented Jul 4, 2024

diegomarquezp commented Jul 5, 2024

JoeWang1127 commented Jul 5, 2024

sonarcloud bot commented Jul 8, 2024

sonarcloud bot commented Jul 8, 2024

	metadata["snippets"] = snippets.all_snippets(
	["samples//src/main/java//.java", "samples/*/pom.xml"]
	)

		)


		def latest_maven_version(group_id: str, artifact_id: str) -> Optional[str]:

chore: transfer java-related synthtool code into the library_generation image #3011

chore: transfer java-related synthtool code into the library_generation image #3011

Conversation

diegomarquezp commented Jul 3, 2024 • edited Loading

Details of code transfer

Changes in Dockerfile

snippet-bot bot commented Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diegomarquezp Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoeWang1127 Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoeWang1127 commented Jul 4, 2024

diegomarquezp commented Jul 5, 2024

JoeWang1127 commented Jul 5, 2024

sonarcloud bot commented Jul 8, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarcloud bot commented Jul 8, 2024

Quality Gate passed for 'java_showcase_integration_tests'

diegomarquezp commented Jul 3, 2024 •

edited

Loading

snippet-bot bot commented Jul 3, 2024 •

edited

Loading

diegomarquezp Jul 4, 2024 •

edited

Loading

JoeWang1127 Jul 4, 2024 •

edited

Loading