-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simplify the build of SWT-binaries and make it more Maven-oriented #513
Comments
First part of eclipse-platform#513 The build procedure is based on the configurations in https://ci.eclipse.org/releng/view/SWT%20Natives/ Namely these are: - https://ci.eclipse.org/releng/job/SWT-Increment_if_needed/configure - https://ci.eclipse.org/releng/job/cocoa_aarch64/configure - https://ci.eclipse.org/releng/job/cocoa_x86_64/configure - https://ci.eclipse.org/releng/job/gtk_linux_aarch64/configure - https://ci.eclipse.org/releng/job/gtk_linux_ppc64le/configure - https://ci.eclipse.org/releng/job/gtk_linux_x86_64/configure - https://ci.eclipse.org/releng/job/win32_x86_64/configure The steps of all those jobs are now perform in the Jenkins-build of this o.e.swt repo and thus makes all the mentioned jobs obsolete. This also makes building the swt-binaries as part of the I-builds obsolete. It has also the advantage that one can get immediate feedback in verification-builds for changes that affect the SWT-binaries. Signed-off-by: Hannes Wellmann <[email protected]>
First part of eclipse-platform#513 The build procedure is based on the configurations in https://ci.eclipse.org/releng/view/SWT%20Natives/ Namely these are: - https://ci.eclipse.org/releng/job/SWT-Increment_if_needed/configure - https://ci.eclipse.org/releng/job/cocoa_aarch64/configure - https://ci.eclipse.org/releng/job/cocoa_x86_64/configure - https://ci.eclipse.org/releng/job/gtk_linux_aarch64/configure - https://ci.eclipse.org/releng/job/gtk_linux_ppc64le/configure - https://ci.eclipse.org/releng/job/gtk_linux_x86_64/configure - https://ci.eclipse.org/releng/job/win32_x86_64/configure The steps of all those jobs are now perform in the Jenkins-build of this o.e.swt repo and thus makes all the mentioned jobs obsolete. This also makes building the swt-binaries as part of the I-builds obsolete. It has also the advantage that one can get immediate feedback in verification-builds for changes that affect the SWT-binaries. Signed-off-by: Hannes Wellmann <[email protected]>
I cannot tell for sure, but I hope so and in case it's not the case, it would be a good target to have it that way ;)
Sounds good.
At the moment, the |
First part of eclipse-platform#513 The build procedure is based on the configurations in https://ci.eclipse.org/releng/view/SWT%20Natives/ Namely these are: - https://ci.eclipse.org/releng/job/SWT-Increment_if_needed/configure - https://ci.eclipse.org/releng/job/cocoa_aarch64/configure - https://ci.eclipse.org/releng/job/cocoa_x86_64/configure - https://ci.eclipse.org/releng/job/gtk_linux_aarch64/configure - https://ci.eclipse.org/releng/job/gtk_linux_ppc64le/configure - https://ci.eclipse.org/releng/job/gtk_linux_x86_64/configure - https://ci.eclipse.org/releng/job/win32_x86_64/configure The steps of all those jobs are now perform in the Jenkins-build of this o.e.swt repo and thus makes all the mentioned jobs obsolete. This also makes building the swt-binaries as part of the I-builds obsolete. It has also the advantage that one can get immediate feedback in verification-builds for changes that affect the SWT-binaries. Signed-off-by: Hannes Wellmann <[email protected]>
At the moment the sources for In the beginning of each I-Build the Jenkins-Job releng/SWT-Increment_if_needed is triggered: If there are changes in o.e.swt, that require recompilation of the SWT-binaries, then this job triggers six other jobs, one to build the swt-binaries for each platform: https://ci.eclipse.org/releng/view/SWT%20Natives/ The main goal of #514 is to perform compilation and committing of the SWT-binaries as part of the We don't loose much in testing here, because at the moment the swt.binaries are build and committed in the very beginning of the I-builds and the tests are performed in the very end. So if there are test-failures the committed binary changes have to be reverted any way. #514 is just moving the binary compilation step from the beginning of the I-builds to the committing process of this repo (which btw. probably also allows to use git-timestamp based qualifier at least for o.e.swt). The extensive testing still happens at the end of the I-builds (just like for other eclipse TLP components as well). If your main concern is to duplicate the agents of the |
What about running SWT pipeline on releng JIPP? That way we should be able to have all benefits you listed without the extra work. |
One more reason to go for releng JIPP is the fact that it is configure to push to git repos which I'm not sure is done for Platform JIPP nor whether it's even desired. |
Yes, that should work as well, I even suggested that already and would be fine with that :)
There might be some extra configuration work necessary to move this repos pipeline to releng JIPP, but it is probably less then setting up the agents. The freeze-period check works without extra credentials/tpkens and tokens for the dash-licenses tools or alike are not necessary in this repo.
That's a point I didn't consider yet, but yes it makes sense. |
If the mentioned Platform SWT jobs do not deploy artifacts to Maven Central... then no. |
They don't. AFAIK SWT is deployed to Maven-Central together with the other parts of eclipse-platform via separate Jobs in the @akurtakov, @fredg02 so do we have consensus to move https://ci.eclipse.org/platform/job/eclipse.platform.swt/ to the If yes, @fredg02 can you perform the move? Should I create a new helpdesk issue for that or can this be done under eclipsefdn/helpdesk#2448? |
Let's move it! |
I can move it, yes. I will actually copy the job and disable it on the platform JIPP until everything works as expected. |
Sounds great. Thank you! I will let you know once #514 is fully implemented and working. |
Thank you again! That was quick 😄 |
I just noticed that all jobs expect the one for my PR #514 are currently starving waiting for an executor. I assume that is because currently the swt-pipeline uses a kubernetes agent: In #514 I changed that to be a agent labeled with |
First part of eclipse-platform#513 The build procedure is based on the configurations in https://ci.eclipse.org/releng/view/SWT%20Natives/ Namely these are: - https://ci.eclipse.org/releng/job/SWT-Increment_if_needed/configure - https://ci.eclipse.org/releng/job/cocoa_aarch64/configure - https://ci.eclipse.org/releng/job/cocoa_x86_64/configure - https://ci.eclipse.org/releng/job/gtk_linux_aarch64/configure - https://ci.eclipse.org/releng/job/gtk_linux_ppc64le/configure - https://ci.eclipse.org/releng/job/gtk_linux_x86_64/configure - https://ci.eclipse.org/releng/job/win32_x86_64/configure The steps of all those jobs are now perform in the Jenkins-build of this o.e.swt repo and thus makes all the mentioned jobs obsolete. This also makes building the swt-binaries as part of the I-builds obsolete. It has also the advantage that one can get immediate feedback in verification-builds for changes that affect the SWT-binaries.
First part of eclipse-platform#513 The build procedure is based on the configurations in https://ci.eclipse.org/releng/view/SWT%20Natives/ Namely these are: - https://ci.eclipse.org/releng/job/SWT-Increment_if_needed/configure - https://ci.eclipse.org/releng/job/cocoa_aarch64/configure - https://ci.eclipse.org/releng/job/cocoa_x86_64/configure - https://ci.eclipse.org/releng/job/gtk_linux_aarch64/configure - https://ci.eclipse.org/releng/job/gtk_linux_ppc64le/configure - https://ci.eclipse.org/releng/job/gtk_linux_x86_64/configure - https://ci.eclipse.org/releng/job/win32_x86_64/configure The steps of all those jobs are now perform in the Jenkins-build of this o.e.swt repo and thus makes all the mentioned jobs obsolete. This also makes building the swt-binaries as part of the I-builds obsolete. It has also the advantage that one can get immediate feedback in verification-builds for changes that affect the SWT-binaries.
@fredg02 the print-out of the build log e.g. of the swt-master is listed below. Could it be that the kubernetes agents request more resources than available? Furthermore I noticed that needed mac agents is not completing its start-up.
|
That's possible.
MacOS agent b9h15-macos10.15 went AWOL and had to be reprovisioned, see also https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2454 |
Is there anything I can do to fix that?
Thanks for that hint. But according to the releng Jenkins overview the mac agent is still launching (since yesterday evening). |
Demand less resources. :)
I've tried to take it offline, but it kept reconnecting. Should be fixed now. I'm trying to get the agent back online today. |
If that is sufficent to compile that SWT-natives for linux-x86_64 that's indeed the simplest solution. :)
Thank you! |
@fredg02 besides the resources issue it looks like the eclipse.platform.swt job on the releng Jenkins is not notified about git-events in this repo. For example when I push a new commit to a PR or even open a new PR the commits/branches are not picked up and build by Jenkins by default and I have to trigger builds manually respectively have to scan the repository manually. Could it be that the webhooks are not configured correctly for the new Jenkins home of this git repos pipeline? |
I have added a webhook to the eclipse.platform.swt repo that triggers the Releng JIPP. Please test it. |
Tested it with a new commit on a existing PR and that works well now. Thanks again. :) |
First part of eclipse-platform#513 The build procedure is based on the configurations in https://ci.eclipse.org/releng/view/SWT%20Natives/ Namely these are: - https://ci.eclipse.org/releng/job/SWT-Increment_if_needed/configure - https://ci.eclipse.org/releng/job/cocoa_aarch64/configure - https://ci.eclipse.org/releng/job/cocoa_x86_64/configure - https://ci.eclipse.org/releng/job/gtk_linux_aarch64/configure - https://ci.eclipse.org/releng/job/gtk_linux_ppc64le/configure - https://ci.eclipse.org/releng/job/gtk_linux_x86_64/configure - https://ci.eclipse.org/releng/job/win32_x86_64/configure The steps of all those jobs are now perform in the Jenkins-build of this o.e.swt repo and thus makes all the mentioned jobs obsolete. This also makes building the swt-binaries as part of the I-builds obsolete. It has also the advantage that one can get immediate feedback in verification-builds for changes that affect the SWT-binaries.
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not useful anymore. Part of eclipse-platform#513 Fixes eclipse-platform#626
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not useful anymore. Part of eclipse-platform#513 Fixes eclipse-platform#626
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not useful anymore. Part of eclipse-platform#513 Fixes eclipse-platform#626
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not useful anymore. Part of eclipse-platform#513 Fixes eclipse-platform#626
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not useful anymore. Part of eclipse-platform#513 Fixes eclipse-platform#626
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not useful anymore. Part of eclipse-platform#513 Fixes eclipse-platform#626
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not useful anymore. Part of eclipse-platform#513 Fixes eclipse-platform#626
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not necessary anymore. Part of eclipse-platform#513 Fixes eclipse-platform#626
This avoids the need for a Java-11 JRE to run Javascript in the pipeline and moves the file changes and git actions into one common place, making it simpler to understand. Simplify the step to commit the binaries binaries by deleting all existing native binaries in the repo before they are build and then just committing all new, deleted and changed files. Also skip version increment tasks that change nothing (i.e. the major- and minor swt-version value are unchanged and thus the entire version.txt is not updated in the automated pipeline. Remove the 'check_fragment_libraries' task without replacement since it is not necessary anymore. Part of #513 Fixes #626
The main work for this issue is now completed.
There are still a few minor things to stream-line or simply thus I leave this open until that is done, but since all ANT scripts are now gone I wanted to share this intermediate update. |
@HannesWell : could you please update wiki https://github.com/eclipse-platform/eclipse.platform.swt/wiki/SWT-Native-Builds-on-Foundation ? Especially I'm interested in "how can one build native SWT fragments locally". |
Yes I will do that soon.
You can build them as part of a local SWT maven build by setting the native property to the platform to build (usually the running one): If you just want to build the natives as fast as possible using maven you can use: If you don't want to use maven running the following two commands from the directory of the project eclipse.platform.swt/binaries/pom.xml Lines 141 to 169 in bd81b48
The exact commands depend on the platform and maybe can also be simplified in the future. Therefore using maven is probably the simplest way. |
Increment the Library revision also for enforced native binary re-builds. Besides allowing simplifications of the build, this ensures there is a git-qualifier update and we don't need to have letter suffixes for git tags. Part of eclipse-platform#513
Increment the Library revision also for enforced native binary re-builds. Besides allowing simplifications of the build, this ensures there is a git-qualifier update and we don't need to have letter suffixes for git tags. Use the org.eclipse.cbi.maven.plugins:eclipse-winsigner-plugin Maven plug-in to sign the built binaries instead of bash-scripts. And remove superfluous steps. Part of eclipse-platform#513
Increment the Library revision also for enforced native binary re-builds. Besides allowing simplifications of the build, this ensures there is a git-qualifier update and we don't need to have letter suffixes for git tags. Use the org.eclipse.cbi.maven.plugins:eclipse-winsigner-plugin Maven plug-in to sign the built binaries instead of bash-scripts. And remove superfluous steps. Part of eclipse-platform#513
An even faster way and a Maven launch configuration that has it encoded is about to be added in #991. |
Increment the Library revision also for enforced native binary re-builds. Besides allowing simplifications of the build, this ensures there is a git-qualifier update and we don't need to have letter suffixes for git tags. Use the org.eclipse.cbi.maven.plugins:eclipse-winsigner-plugin Maven plug-in to sign the built binaries instead of bash-scripts. And remove superfluous steps. Part of eclipse-platform#513
Increment the Library revision also for enforced native binary re-builds. Besides allowing simplifications of the build, this ensures the git-based qualifier of the org.eclipse.swt and consequently its native fragments (which use the same qualifier) is updated in that case and a subsequently 'manually' enforced qualifier update is not necessary. Furthermore letter-suffixes for revision git tags are not necessary too. Use the org.eclipse.cbi.maven.plugins:eclipse-winsigner-plugin Maven plug-in to sign the built binaries instead of bash-scripts. And remove superfluous steps. Part of eclipse-platform#513
Increment the Library revision also for enforced native binary re-builds. Besides allowing simplifications of the build, this ensures the git-based qualifier of the org.eclipse.swt and consequently its native fragments (which use the same qualifier) is updated in that case and a subsequently 'manually' enforced qualifier update is not necessary. Furthermore letter-suffixes for revision git tags are not necessary too. Remove superfluous steps. Part of eclipse-platform#513
Increment the Library revision also for enforced native binary re-builds. Besides allowing simplifications of the build, this ensures the git-based qualifier of the org.eclipse.swt and consequently its native fragments (which use the same qualifier) is updated in that case and a subsequently 'manually' enforced qualifier update is not necessary. Furthermore letter-suffixes for revision git tags are not necessary too. Remove superfluous steps. Part of eclipse-platform#513
Increment the Library revision also for enforced native binary re-builds. Besides allowing simplifications of the build, this ensures the git-based qualifier of the org.eclipse.swt and consequently its native fragments (which use the same qualifier) is updated in that case and a subsequently 'manually' enforced qualifier update is not necessary. Furthermore letter-suffixes for revision git tags are not necessary too. Remove superfluous steps. Part of #513
Remove requirement to specify the arch as first argument for Windows in the build.bat. The first argument always has to be 'x86_64' but the value is then never used. Part of eclipse-platform#513
Remove requirement to specify the arch as first argument for Windows in the build.bat. The first argument always has to be 'x86_64' but the value is then never used. Part of eclipse-platform#513
Remove requirement to specify the arch as first argument for Windows in the build.bat. The first argument always has to be 'x86_64' but the value is then never used. Part of #513
eclipse.platform.swt/binaries/pom.xml Line 50 in 2378112
Typo: |
That profile is not activated explicitly, only automatically where needed, so it would not be dramatic. Nevertheless thanks for spotting this, just created #1141 to fix it. |
--> #1167 |
Discussed in #497
Originally posted by HannesWell December 11, 2022
The build of the SWT-binaries is a quite complex process and I have the impression that it is even more complex than it would have to be, from the fact that o.e.swt contains the sources for all platforms but the build produces a fragment for each platform that contains all, the common and platform specific compiled code and native libraries.
I'm just about the learn the entire build-process of SWT (because of #490), so please correct me if I'm wrong or missing something.
One thing that makes it IMHO complex is the usage of multiple different ant-scripts that copy sources multiple times, back and fourth and that it looks like that the java-code is compiled multiple times and that besides the standard main- and source-jar there is also a zip produced for each platform-specific fragment.
The zip seems to be the one that is presented in the SWT section of the I-build drops:
https://download.eclipse.org/eclipse/downloads/drops4/I20221209-1800/
Is it correct that the
o.e.swt
and platform specific swt fragments that end up in the eclipse-updates repos (like https://download.eclipse.org/eclipse/updates/4.27-I-builds/) are the regular main- and sources-jar of the corresponding maven project?I wonder why the code is compiled twice? And from my understanding of the build-process, as described above, this means that the class-files in the jars that end up in the p2-repo are different ones than those presented as swt-zips in the drops.
In general I think that it would be good if the swt.binaries build would be more maven oriented and I think it could be just like:
This would also include that all ant based launch configs or procedures are replaced by ones that include executing a Maven build.
A second bigger point that could be improved is where/when the native-libs in each platform-specific fragment are build.
At the moment this happens in https://ci.eclipse.org/releng/job/SWT-Increment_if_needed, which is triggered in the beginning of the daily releng/I-build job.
I wonder if it would be better to migrate that job to steps that run as part of the
master
-branch build of this repo? The build of the native libraries (if necessary) could even be done in the beginning of the master-build so that the following verification always runs on the native libs that match it. At the moment (as far as I understand) the master branch is build against the native libs build in the last I-build. If the native libs have changed the changed lib could be committed to the swt.binaries repo in the end if the master-build succeeded.This would allow to get rid of the
SWT-Increment_if_needed
and if we would use https://www.jenkins.io/doc/book/pipeline/syntax/#parallel maybe even all the platform-specific jenkins job could be removed as well (but I have not yet checked all).This would also allow to remove the enfored qualifier for the o.e.swt Plugin:
eclipse.platform.swt/bundles/org.eclipse.swt/pom.xml
Line 32 in 1d81184
It could then just use the qualifier based on the git-timestamp, just like most other Plugins do it. If that qualifier ahd changed it could then be set and committed to the
o.e.swt.binaries/binaries-parent/pom.xml
, together with the changed native-libs/binaries in the end of this repos master-build.In the I-builds pipeline the
Swt build input
stage could then be removed entirely.@akurtakov, @mickaelistria, what do you think?
Maybe @SDawley or @niraj-modi are you interested in that as well?
The text was updated successfully, but these errors were encountered: