Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace content of o.e.osgi.services by osgi-bundles from Maven-Central #44

Closed

Conversation

HannesWell
Copy link
Member

As discussed in eclipse-equinox/equinox#18, this replaces the embedded osgi-sources of org.eclipse.osgi.services by corresponding osgi-bundles from Maven-Central.

As a side effect this updates the versions of

  • org.osgi.service.component.* 1.4.0 -> 1.5.0

To comply with the Eclipse deprecation/removal process org.eclipse.osgi.services is not removed entirely but it's content is replaced by requiring and re-exporting the provided osgi-bundles from Maven-Central. Bundles that require-bundle org.eclipse.osgi.services directly continue to work.

@HannesWell
Copy link
Member Author

This again requires eclipse-platform/eclipse.platform.releng.aggregator#235 to be submitted first.

Comment on lines 12 to 13
Import-Package: org.osgi.annotation.versioning;version="[1.1.2,1.2.0)",
org.osgi.service.component.annotations;version="[1.3,1.4)"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want runtime import of compile time packages. No bundle exports org.osgi.annotation.versioning. We also don't need to import org.osgi.service.component.annotations since this bundle does not use it in any way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagine the versioning package is imported here to allow the build to succeed for the remaining source in this bundle. Before we put some project JARs on the project classpath to do this. We could make this import on versioning optional or go back to a project JAR or some other solution.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagine the versioning package is imported here to allow the build to succeed for the remaining source in this bundle. Before we put some project JARs on the project classpath to do this. We could make this import on versioning optional or go back to a project JAR or some other solution.

That was indeed the reason. I was too fast in removing it and then tried to resolve the compile errors and forgot to fix that afterwards. Thanks for the hint!
I think going back to the jar is the most robust solution, isn't it? And since the days of this bundle are numbered, I would not worry to much in finding a more elegant solution.

Regarding the import of org.osgi.service.component.annotations: I suspect the import was mainly because this bundle is exporting that package? But I removed it now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it is plainly wrong to have an Import-Package in the resulting binary just to make sure source code can compile. This is an example of how PDE conflates build configuration and build output in the Manifest.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restoring use of the osgi.annotation binary solves the need for the Import-Package but there should be some other way to get osgi.annotation on the compile path.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn’t there an extra classpath property you can put in the build.properties for such build/compile time dependencies? I think Tycho respects that too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is jars.extra.classpath = lib/osgi.annotation.jar but this now refers to a local binary in the project source. It would be better if it could refer to some artifact in a repository like maven central.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine to have as an incremental improvement. We wouldn't need any of this if we got rid of the org.eclipse.osgi.services bundle altogether. But that takes a longer announcement and deprecation process and even then we will likely get loud objections to the removal. The other option is to simply have the class files sucked into the produced JAR during the build from the available maven artifacts. The disadvantage of that is that this will not work for the PDE project imported into the workspace.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with everything said in this thread. To me it looks like the root of this problem is that PDE does not support different scopes of dependencies, for example like Maven does. I don't think this is a problem of OSGi because is 'just' the runtime and its PDEs job to assemble a suitable runtime respectively compile-time reactor/container.
PDE uses the same set of dependencies when resolving the PDE-State (the 'Compile-Container') and when launching an Eclipse-Application (the 'Runtime-Container'). But it could use different sets (e.g. compile-time only like for the annotations). To avoid the need ot add the org.osgi.service.component.annotations when using OSGi DS those annotations are treated specially by PDE. If one enables a corresponding setting the annotation-jar with desired version is added implicitly to the classpath at compile-time. So basically PDE already has different Sets of dependencies but in for a very special scenario. If this concept where generalized it could also avoid the need to embed the service.component.annotation jars in PDE.
There could also be another scope for tests (about which I recently had a chat with @laeubi).

However this would be a completely different story (but I think an interesting one) and way out of scope of this change.
Therefore I would like to follow Tom's suggestion and proceed for now with the smaller steps we can do now :) ... if the updated SDK-prereq-TP is published by the platform.aggr build, which fails due to a subsequent change.

bundles/org.eclipse.osgi.services/META-INF/MANIFEST.MF Outdated Show resolved Hide resolved
bundles/org.eclipse.osgi.services/pom.xml Outdated Show resolved Hide resolved
@HannesWell
Copy link
Member Author

HannesWell commented Apr 27, 2022

The Target-Resolution fails due to the following reasons (the package javax.servlet [2.1.0,3.0.0) cannot be found):

[INFO] Resolving dependencies of MavenProject: org.eclipse.osgi:org.eclipse.osgi.services:3.11.0-SNAPSHOT @ /home/jenkins/agent/workspace/rt.equinox.framework_PR-44/bundles/org.eclipse.osgi.services/pom.xml
[INFO] {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86_64}
[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.eclipse.osgi.services 3.11.0.qualifier
[ERROR]   Missing requirement: org.osgi.service.http 1.2.2.202109301733 requires 'java.package; javax.servlet [2.1.0,3.0.0)' but it could not be found
[ERROR]   Cannot satisfy dependency: org.eclipse.osgi.services 3.11.0.qualifier depends on: osgi.bundle; org.osgi.service.http [1.2.0,1.3.0)

The Maven-Metadata of org.osgi.service.http do not specify a dependency to a corresponding Artifact so Tycho is not to blame here.
But for example the jetty-servlet-api BSN org.eclipse.jetty.servlet-api bundle, which is in the target-platform, exports that package but in version 4.0.0.
According to the publication-dates of javax.servlet:javax.servlet-api I derive that javax.servlet [2.1.0,3.0.0) is a relative old package and I wonder if the version-range of the import in org.osgi.service.http should be updated or at least relaxed to [2.1,5.0)?
The sources of org.osgi.service.http currently embedded in Equinox are compiled successfully against javax.servlet 4.0.

So this could either be adjusted in the osgi-bundles (but I assume this will take some time) or we could add javax.servlet<3.0 to the SDK-prereq-TP too.

@tjwatson
Copy link
Contributor

So this could either be adjusted in the osgi-bundles (but I assume this will take some time) or we could add javax.servlet<3.0 to the SDK-prereq-TP too.

We could look to use org.apache.felix:org.apache.felix.http.servlet-api:1.2.0 instead for the servlet APIs. They do some trickery to export the packages from Servlet 4.0 at different versions. It is a bit of a mess, but may help us work here. This is the content if that bundles Export-Package header:

Export-Package: 
  javax.servlet;version="2.6";uses:="javax.servlet.annotation,javax.servlet.descriptor",
  javax.servlet.annotation;version="2.6";uses:="javax.servlet",javax.servlet.descriptor;version="2.6",
  javax.servlet.http;version="2.6";uses:="javax.servlet",
  javax.servlet;version="3.0",
  javax.servlet.annotation;version="3.0",
  javax.servlet.descriptor;version="3.0",
  javax.servlet.http;version="3.0",
  javax.servlet;version="3.1",
  javax.servlet.annotation;version="3.1",
  javax.servlet.descriptor;version="3.1",
  javax.servlet.http;version="3.1",
  javax.servlet;version="4.0",
  javax.servlet.annotation;version="4.0",
  javax.servlet.descriptor;version="4.0",
  javax.servlet.http;version="4.0"

That would also pull in a capability we have been missing for the osgi.contract named JavaServlet

Provide-Capability: osgi.contract;osgi.contract=JavaServlet;version:Li
 st<Version>="2.6,3.0,3.1,4.0";uses:="javax.servlet,javax.servlet.http
 ,javax.servlet.descriptor,javax.servlet.annotation"

@rotty3000 Does this sound like a good option?

@tjwatson
Copy link
Contributor

For folks who are paying attention to the finer details. I opened https://issues.apache.org/jira/browse/FELIX-6525 about the fact that the exports for the various servlet packages does not contain the proper uses directives.

@rotty3000
Copy link
Contributor

Hey @tjwatson i'd be ok with using the Felix API bundle. We're using it in our product with equinox.http.servlet anyway.

HannesWell added a commit to HannesWell/eclipse.platform.releng.aggregator that referenced this pull request Apr 28, 2022
HannesWell added a commit to HannesWell/eclipse.platform.releng.aggregator that referenced this pull request Apr 28, 2022
@HannesWell
Copy link
Member Author

HannesWell commented Apr 28, 2022

So this could either be adjusted in the osgi-bundles (but I assume this will take some time) or we could add javax.servlet<3.0 to the SDK-prereq-TP too.

We could look to use org.apache.felix:org.apache.felix.http.servlet-api:1.2.0 instead for the servlet APIs. They do some trickery to export the packages from Servlet 4.0 at different versions. It is a bit of a mess, but may help us work here. This is the content if that bundles Export-Package header:

So we should simply add org.apache.felix:org.apache.felix.http.servlet-api:1.2.0 to the TP like prepared in eclipse-platform/eclipse.platform.releng.aggregator#239?

Should we then remove jakarta.servlet:jakarta.servlet-api:4.0.4?
There a third provider of the javax.servlet package in the TP: org.eclipse.jetty.toolchain:jetty-servlet-api:4.0.6 probably some transitive dependency of one of the jetty-artifacts.

Or should we wait until FELIX-6525 is fixed and released to avoid the problems you have described?

Relaxing the Package-Import version-range in the osgi-bundle is not an option? I have not yet thought much about it, but my first thought is that a a large as possible version range on the consumer-site gives the greatest flexibility to the resolver. Of course the consumer then must only use the capabilities of the lowest version required and higher major versions then must not break the part of the API that is used.

Besides that I suspect that we also have to replace org.apache.felix.scr_2.1.24.v20200924-1939 from Orbit by org.apache.felix:org.apache.felix.scr:2.2.0 since the former requires org.osgi.service.component-1.4. The good news is that the BSN in Orbit is the same like in Maven-Central. So from that point it could be a drop in replacement.

@HannesWell
Copy link
Member Author

After I adding org.apache.felix:org.apache.felix.http.servlet-api:1.2.0 to the Eclipse-SDK-prerequ TP and referencing that locally my local build fails a bit later with the following error:

[INFO] Adding repository https://download.eclipse.org/eclipse/updates/4.24-I-builds
[INFO] Resolving dependencies of MavenProject: org.eclipse.osgi:org.eclipse.osgi:3.18.0-SNAPSHOT @ C:\dev\git\equinox.framework\bundles\org.eclipse.osgi\pom.xml
[INFO] Resolving dependencies of MavenProject: org.eclipse.osgi:org.eclipse.osgi.compatibility.state:1.2.600-SNAPSHOT @ C:\dev\git\equinox.framework\bundles\org.eclipse.osgi.compatibility.state\pom.xml
[INFO] Resolving dependencies of MavenProject: org.eclipse.osgi:org.eclipse.osgi.services:3.11.0-SNAPSHOT @ C:\dev\git\equinox.framework\bundles\org.eclipse.osgi.services\pom.xml
[INFO] {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86_64}
[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.eclipse.osgi.services 3.11.0.qualifier
[ERROR]   Missing requirement: org.osgi.service.http.whiteboard 1.1.1.202109301733 requires 'osgi.contract; (&(osgi.contract=JavaServlet)(version=3.1.0))' but it could not be found
[ERROR]   Cannot satisfy dependency: org.eclipse.osgi.services 3.11.0.qualifier depends on: osgi.bundle; org.osgi.service.http.whiteboard [1.1.0,1.2.0)
[ERROR] 
[ERROR] See https://wiki.eclipse.org/Tycho/Dependency_Resolution_Troubleshooting for help.
[ERROR] Cannot resolve dependencies of MavenProject: org.eclipse.osgi:org.eclipse.osgi.services:3.11.0-SNAPSHOT @ C:\dev\git\equinox.framework\bundles\org.eclipse.osgi.services\pom.xml: See log for details -> [Help 1]

So it look like the osgi.contract=JavaServlet you mentioned, which is required by org.osgi.service.http.whiteboard still causes problems although it is actually provided by org.apache.felix.http.servlet-api.
The Manifest of felix.https.servlet-api contains:

Provide-Capability: osgi.contract;osgi.contract=JavaServlet;version:List<Version>="2.6,3.0,3.1,4.0";uses:="javax.servlet,javax.servlet.http,javax.servlet.descriptor,javax.servlet.annotation"

And the Manifest of org.osgi.service.http.whiteboard contains:

Require-Capability: osgi.contract;osgi.contract=JavaServlet;filter:="(&(osgi.contract=JavaServlet)(version=3.1.0))",osgi.ee;filter:="(&(osgi.ee=JavaSE/compact1)(version=1.8))"

I wonder if this is a configuration issue or maybe a bug in Tycho?

@HannesWell
Copy link
Member Author

Provide-Capability: osgi.contract;osgi.contract=JavaServlet;version:List<Version>="2.6,3.0,3.1,4.0";uses:="javax.servlet,javax.servlet.http,javax.servlet.descriptor,javax.servlet.annotation"

From my current debugging I have the impression that P2 or Tycho does not read the version list correctly. Because it creates only one ProvidedCapability with namespace osgi.contract and version 0.0.0.

@HannesWell
Copy link
Member Author

HannesWell commented May 3, 2022

Looks like the problem is in org.eclipse.equinox.p2.publisher.eclipse.BundlesAction.addCapability(), which does handle handle List of versions. Instead it creates the empty version if the version-attribute's value is not a Version object. I think it should create a capability for each version. I will prepare a PR tomorrow.

@laeubi
Copy link
Member

laeubi commented May 4, 2022

I wonder if this is a configuration issue or maybe a bug in Tycho?

Just to be clear (and you have already found that out) Tycho do not actually do any resolving/handling here but delegates that work to P2!

@HannesWell
Copy link
Member Author

I wonder if this is a configuration issue or maybe a bug in Tycho?

Just to be clear (and you have already found that out) Tycho do not actually do any resolving/handling here but delegates that work to P2!

That's right.
I created eclipse-equinox/p2#64 to address that issue and hopefully can complete it tomorrow.

@tjwatson
Copy link
Contributor

If we live with representing each version as a separate capability in the short-term, is there any path forward that would allow us to fix that in the future? I worry about pushing such things so late in the release.

@merks
Copy link
Contributor

merks commented May 25, 2022

It seems quite challenging to evolve this. With requirements we had IRequirement and IRequiredCapability so we could introduce something new (which had a negative impact on Oomph and the Aggregator). But with IProvidedCapability it seems harder to evolve. One possibility though is that the IProvidedCapability's getProperties could encode additional information and that's already recorded by MetadataWriter.writeProvidedCapability(IProvidedCapability). Perhaps a boolean flag/property that indicates "I am a singleton capability so if there are other capabilities with the same name and namespace, only one of us may be used to resolve a RequiredPropertiesMatch". I have no idea how to implement the latter though. Perhaps such an approach could be used to fix the broken osgi.ee stuff by making the "singleton property" default to true for certain known namespaces...

@HannesWell
Copy link
Member Author

HannesWell commented May 25, 2022

If we live with representing each version as a separate capability in the short-term, is there any path forward that would allow us to fix that in the future? I worry about pushing such things so late in the release.

Are you referring to completing this PR?
If so, the answer is yes: In conjunction with eclipse-platform/eclipse.platform.releng.aggregator#239 and maybe an update of felix.scr this can be completed. I have already verified that with a locally build version of p2 and Tycho.
But since at the moment we only publish P2 to Maven-Central on releases we have to wait until the Eclipse 2022-06 release any way until we can proceed with this one. Unless we do eclipse-equinox/p2#23 (which I would be in favour of for cases like this one) we could not proceed with this change before 2022-09 has been released, if we wait for a better solution until the next release.

If you were referring to fix the change suggested in eclipse-equinox/p2#64, then we could consider to change the model to have a List/Set of versions in ProvidedCapability, as suggested there. But this would require more work I will definitely not complete today. And, I have to check it, but if the multi-version capabilities are written to p2-repo metadata, old versions of p2 are likely not able to consume new p2-repos any more.

Nevertheless if we continue with 'unrolled' single-version capabilities the 'matcher' could be enhanced to assemble the single version capabilities back into a multi-versioned one by merging all capabilities from the same provider with same namespace and properties. I think that should be distinct, shouldn't it?
Or is there a difference in:

Provide-Capability: 
 cap1;cap1=name3;version:List<Version>="1,2.1";uses:="some.other"

and

Provide-Capability: 
 cap1;cap1=name1;version:Version="1";uses:="some.other",
 cap1;cap1=name1;version:Version="2.1";uses:="some.other"

Which would then be treated equivalent.

But I think we should discuss this in eclipse-equinox/p2#64

@HannesWell HannesWell force-pushed the osgiServicesPackages branch 2 times, most recently from 0c49136 to 4870d9a Compare May 26, 2022 20:06
@HannesWell
Copy link
Member Author

As discussed in eclipse-equinox/p2#64 (comment) I reverted the removal of the embedded sources of the org.osgi.service.http.whiteboard and org.osgi.service.http.whiteboard bundles because both require the capabilities provided by org.apache.felix.http.servlet-api, which cannot be build at the moment due to eclipse-equinox/p2#64.

I would like to complete this as soon as the master-branch re-opens for the 2022-09 release cycle.

@HannesWell HannesWell added this to the 4.25 M1 milestone May 26, 2022
org.osgi.service.upnp;version="1.2",
org.osgi.service.useradmin;version="1.1";uses:="org.osgi.framework",
org.osgi.service.wireadmin;version="1.0.1";uses:="org.osgi.framework"
org.osgi.service.log;version="1.5";uses:="org.osgi.framework"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could get rid of the actual classes from the org.osgi.service.log package by placing a mandatory
matching attribute on the package.

  org.osgi.service.log;version="1.5";uses:="org.osgi.framework";mandatory:="never.ever.use";never.ever.use="true"

If I recall correctly, at least for the Equinox framework implementation, we consider the Import-Package of a package before allowing its export to be selected. So if a bundle has an Import-Package which can never match its own export then its export will always be dropped. Here the import would never be satisfied by our own export because of the mandatory directive for "never.ever.use". This implies that the empty content from org.eclipse.osgi.services for the org.osgi.service.log package could never be used because it would always be substituted by the import to something real.

We would have to validate that is true.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an interesting suggestion. I will try and validate that tomorrow.
It would be great if we can delete the service.log package too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implies that the empty content from org.eclipse.osgi.services for the org.osgi.service.log package could never be used because it would always be substituted by the import to something real.

We would have to validate that is true.

I just tested it with a test project that contained Require-Bundle: org.eclipse.osgi.services in its manifest and and uses the Logger interface from org.osgi.service.log. The suggested approach worked well with PDE, when build with Tycho and when launched into an Eclipse application. The classes from org.osgi.service.log were available to the compiler respectively could be loaded at runtime. At least at compile-time the mandatory was not necessary. But I think it is better to be save.

I removed the embedded sources from org.osgi.service.log but added a not to not have a empty directory to silence PDE.

@HannesWell
Copy link
Member Author

To really use org.osgi.service.component in version 1.5.0 org.apache.felix.scr has to be updated to version 2.2.0 in the eclipse-sdk-prereqs.target. I will prepare a corresponding PR for platform.aggregator and suggest we use the version from Maven-Central directly.

@tjwatson
Copy link
Contributor

To really use org.osgi.service.component in version 1.5.0 org.apache.felix.scr has to be updated to version 2.2.0 in the eclipse-sdk-prereqs.target. I will prepare a corresponding PR for platform.aggregator and suggest we use the version from Maven-Central directly

We cannot really do that until I do a proper release of Felix scr to have it import the osgi packages. I'll try to get that started tomorrow.

@tjwatson
Copy link
Contributor

To be more clear we need this commit to be released apache/felix-dev@bb03476

@HannesWell
Copy link
Member Author

To really use org.osgi.service.component in version 1.5.0 org.apache.felix.scr has to be updated to version 2.2.0 in the eclipse-sdk-prereqs.target. I will prepare a corresponding PR for platform.aggregator and suggest we use the version from Maven-Central directly

We cannot really do that until I do a proper release of Felix scr to have it import the osgi packages. I'll try to get that started tomorrow.

To be more clear we need this commit to be released apache/felix-dev@bb03476

Roger that.

Since we have to wait for the Eclipse release we have some time any way.

Since we need it when we want to remove the http packages as well, do you plan to release the change apache/felix-dev@3a9b563 in org.apache.felix.http.servlet-api soon as well? Or will this be part of the release you already plan?

@HannesWell
Copy link
Member Author

To really use org.osgi.service.component in version 1.5.0 org.apache.felix.scr has to be updated to version 2.2.0 in the eclipse-sdk-prereqs.target. I will prepare a corresponding PR for platform.aggregator and suggest we use the version from Maven-Central directly

We cannot really do that until I do a proper release of Felix scr to have it import the osgi packages. I'll try to get that started tomorrow.

Although the release is not yet available I already created the corresponding PR: eclipse-platform/eclipse.platform.releng.aggregator#282
I will update the versions accordingly as soon as the next release is available.

And update to latest version of
- org.osgi.service.cm
- org.osgi.service.component
- org.osgi.service.device
- org.osgi.service.event
- org.osgi.service.metatype
- org.osgi.service.provisioning
- org.osgi.service.upnp
- org.osgi.service.useradmin
- org.osgi.service.wireadmin

Discussed in issue:
https://github.com/eclipse-equinox/equinox.framework/issues/40
@HannesWell
Copy link
Member Author

Since the prerequisite for this PR, eclipse-platform/eclipse.platform.releng.aggregator#282, will probably be submitted soon: @tjwatson and @bjhargrave could you please renew your review on this PR so that we can submit this as soon all prerequisites are available.

Furthermore I build this change locally and got the following test-failure:

org.eclipse.osgi.tests.bundles.SystemBundleTests.testBug405919  Time elapsed: 0.025 s  <<< FAILURE!
java.lang.AssertionError: Number of capabilities differ expected:<277> but was:<349>
	at org.eclipse.osgi.tests.bundles.SystemBundleTests.testBug405919(SystemBundleTests.java:2079)

Do you have an idea why this happens? Maybe this is related to the removal of the log-packages. Or maybe it is just due to the different setup.

Copy link
Member

@laeubi laeubi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We recently moved the repository content to https://github.com/eclipse-equinox/equinox would you be so kind to open your PR for the new repository?

@HannesWell
Copy link
Member Author

We recently moved the repository content to https://github.com/eclipse-equinox/equinox would you be so kind to open your PR for the new repository?

Migrated to eclipse-equinox/equinox#30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants