Replace embedded service.http sources in org.eclipse.osgi.services by original bundles and deprecate o.e.osgi.services

[HannesWell]

This replaces the remaining embedded org.osgi.service.http* packages in org.eclipse.osgi.services by the original bundles from Maven-Central and then marks org.eclipse.osgi.services as deprecated.
Fixes #18.

As suggested in eclipse-platform/eclipse.platform.releng.aggregator#239 (comment):

This is a sort of hack, but to use the existing jakarta.servlet:jakarta.servlet-api artifact could we also deploy a fragment to jakarta.servlet:jakarta.servlet-api that adds the osgi.contract capability we need? If I recall correctly that was the missing piece to allow the osgi JARs to work for providing the http service APIs.

This adds a fragment org.eclipse.osgi.service.http.servlet.contract to supply the capability: osgi.contract;osgi.contract=JavaServlet, required by org.osgi.service.http.whiteboard to the known javax.servlet providing OSGi bundles from Maven-Central:

• javax.servlet:javax.servlet-api
• jakarta.servlet:jakarta.servlet-api:[4.0.2-4.0.4]
• org.eclipse.jetty.toolchain/jetty-servlet-api

javax.servlet:servlet-api does not provide OSGi metadata.

One non-optimal thing with the different javax.servlet provides (except the ones from jetty) is that in the version 4 stream all artifacts have version 4.0 regardless of the BSN or their Maven-version change.
However after a quick/coarse check it looks like they only differences are white-space or legal text changes.

Background about OSGi contracts: https://docs.osgi.org/reference/portable-java-contracts.html

Another greater problem is that org.osgi.service.http has in its Manifest:

Import-Package:
 javax.servlet;version="[2.1,3)",
 javax.servlet.http;version="[2.1,3)"

In order to fix that I also added corresponding exports of javax.servlet(.http) in version 2.999 to org.eclipse.osgi.service.http.servlet.contract that are served by the host. I compared all of the javax.servlet providing artifacts mentioned above and all content in javax.servlet or javax.servlet.http contained in javax.servlet:servlet-api:2.2 is also provided by the other providers.

[HannesWell]

The real provider has

Suggested change

	javax.servlet;version="2.999",
	javax.servlet;version="2.999";uses:="javax.servlet.annotation,javax.servlet.descriptor",

But the two used packages do not exist in javax.servlet 2.2.

[HannesWell]

Applied this now.
Additionally used packages should not harm consumers of the servlet version 2 package since they can't use it.

[HannesWell]

According to https://docs.osgi.org/reference/portable-java-contracts.html the osgi.contract=JavaServlet is specified with version 2.5, 3.0, 3.1 and 4.0.
But org.apache.felix:org.apache.felix.http.servlet-api:1.2.0 defines 2.6 instead of 2.5.

[HannesWell]

Changed the list to have the official version. Since o.o.service.http.whiteboard only uses 3.1 all the others doesn't matter for our use-case anyway.

[HannesWell]

Should we support only javax/jakarta.servlet-apior also jetty-servlet-api?

[github-actions]

Test Results

     24 files  ±0       24 suites  ±0   11m 24s ⏱️ -1s
2 150 tests ±0  2 105 ✔️ ±0  45 💤 ±0  0 ❌ ±0 
2 194 runs  ±0  2 149 ✔️ ±0  45 💤 ±0  0 ❌ ±0 

Results for commit 258a200. ± Comparison against base commit 9658936.

♻️ This comment has been updated with latest results.

[HannesWell]

In general to discuss the path forward we should also consider @tjwatson remarks from eclipse-platform/eclipse.platform.releng.aggregator#239 (comment):

I also wonder how long will Equinox will have to support javax.servlet?

That is a complicate question.

1. there is no http service spec that will be updated to the jakarta namespace, only the servlet whiteboard spec is updated to jakarta.  So we have to deprecate and remove the implementation of the http service implementation

2. the javax.servlet API is in other APIs of ours like the http.registry.  So we have to deprecate some of that and replace parts that use jakarta namespace

3. all current users of javax.servlet have to migrate their code to jakarta also

These are not things we can easily do. And that is just from a deprecation breaking POV. I've not mentioned the actual work of changing our own codebase to use jakarta and who will be dedicated to doing that work.

[HannesWell]

The build has problems to resolve the javax.servlet package exported by the fragment, but also in the IDE workspace (at least after a restart) the import
Import-Package: javax.servlet;version="[2.1,3)" is not resolved while Import-Package: javax.servlet.http;version="[2.1,3)" is.
I'm puzzled why this is the case.

Maybe it would just be simpler to use org.apache.felix:org.apache.felix.http.servlet-api:1.2.0 as suggested in
eclipse-platform/eclipse.platform.releng.aggregator#239.

Another alternative @laeubi suggested (in a private chat), to leverage the missing metadata of javax.servlet:servlet-api:2.5 and just generate them suitably using the new Orbit SimRel infrastructure created by Ed. The instructions could be prototyped in the sdk-target.

[laeubi]

@HannesWell thanks for the efforts and detailed analysis.

I think it makes no sense to use an "official" artifact if when we need extra efforts to use it like additional fragments and alike, this defeats the purpose of it.

For me the proposal made here to reuse the felix bundled version:

• Add org.apache.felix:org.apache.felix.http.servlet-api:1.2.0 eclipse-platform/eclipse.platform.releng.aggregator#239

seem much more appropriate because of the following reasons:

1. It is proven to work in OSGi and the felix-http has a very active and vibrant development and is the de factor standard
2. It is very unlikely that Equinox will ever get a comparable implementation, so as a matter of facts we should go the way we did with DS and try to migrate Equinox/Platform to the felix impl, using their servlet API package seems like a first simple and non intrusive step forward.

WDYT @akurtakov @tjwatson ?

[HannesWell]

I think it makes no sense to use an "official" artifact if when we need extra efforts to use it like additional fragments and alike, this defeats the purpose of it.

For me the proposal made here to reuse the felix bundled version:

* [Add org.apache.felix:org.apache.felix.http.servlet-api:1.2.0 eclipse-platform/eclipse.platform.releng.aggregator#239](https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/pull/239)

seem much more appropriate because of the following reasons:

1. It is proven to work in OSGi and the felix-http has a very active and vibrant development and is the de factor standard

2. It is very unlikely that Equinox will ever get a comparable implementation, so as a matter of facts we should go the way we did with DS and try to migrate Equinox/Platform to the felix impl, using their servlet API package seems like a first simple and non intrusive step forward.

Yes, with all the issues that arose on the way (the CI and WS build is still failing/has issues a.t.m.), I'm more and more inclined towards that solution as well. Especially, as you said, the two specs have probably not a big future.

[HannesWell]

@tjwatson could you please share your assessment on how we should proceed here?

[tjwatson]

@tjwatson could you please share your assessment on how we should proceed here?

I think migrating to the felix http/whiteboard implementation sounds interesting. But it has a myriad of issues with respect to extensions our implementation has on top of the spec that we either have to deprecate and eventually remove (before we can use the felix imple) or we have to figure out how to implement them on top of the felix implemetnation.

I think we should first and foremost focus on how to replace the service.http sources from the osgi.services bundle.

Perhaps we just add the capability osgi.contract;osgi.contract=JavaServlet to the org.eclipse.equinox.http.servlet bundle since it imports the necessary servlet packages so can ensure the contract is there? That should allow the OSGi artifact for http.service to resolve.

[laeubi]

I just wanted to note that org.osgi.service.http does NOT require the contract at all, it is the org.osgi.service.http.whiteboard what is a different service and as far as I know was never provided by equinox.

[tjwatson]

I just wanted to note that org.osgi.service.http does NOT require the contract at all, it is the org.osgi.service.http.whiteboard what is a different service and as far as I know was never provided by equinox.

We do provide an implementation of org.osgi.service.http.whiteboard in org.eclipse.equinox.http.servlet

[HannesWell]

@tjwatson could you please share your assessment on how we should proceed here?

I think migrating to the felix http/whiteboard implementation sounds interesting. But it has a myriad of issues with respect to extensions our implementation has on top of the spec that we either have to deprecate and eventually remove (before we can use the felix imple) or we have to figure out how to implement them on top of the felix implemetnation.

I don't want to and cannot judge about replacing the implementation, but this PR is only about replacing the embedded http/whiteboard API sources with the OSGi bundles from bundles from Maven-Central.

But since the original http/whiteboard OSGi bundles have some difficult dependencies (as desribed in the initial comment of this PR), the question is how to provide them.

This PR initially suggested to add a fragment to equinox that adds the required capabilities to one of the 'original' javax.servlet artifacts, but at the moment I'm more inclined to use the org.apache.felix.http.servlet-api:1.2 which provides the classes from these javax.servlet artifacts but additionally with the missing metadata.
So it would be eclipse-platform/eclipse.platform.releng.aggregator#239 plus this PR without the addition of the new fragment.

[tjwatson]

A fragment is only allowed to specify a single host.

[HannesWell]

Ok, somehow I assume this is possible. At least the runtime didn't complain, but looking in the spec it does not look like this is permitted.
But ok, then we would have to select one of these three, in case we continue with adding this fragment. In that case it is probably the most future-proof to choose jakarta.servlet-api.

[laeubi]

@tjwatson are you sure? I also though this but remember I have seen such "multi-host" already somewhere else (sadly I can't remember where/when) and was surprised there as well...

[tjwatson]

@tjwatson are you sure? I also though this but remember I have seen such "multi-host" already somewhere else (sadly I can't remember where/when) and was surprised there as well...

Yes, I'm sure the spec Fragment-Host syntax only allows for a single host symbolic name to be defined:

https://docs.osgi.org/specification/osgi.core/8.0.0/framework.module.html#framework.module.fragmenthost

Fragment-Host       ::= bundle-description
bundle-description  ::= symbolic-name 
                            ( ';' parameter )* // See [1.3.2](https://docs.osgi.org/specification/osgi.core/8.0.0/framework.introduction.html#framework.general.syntax)

It is true a fragment may be attached to multiple host bundles, but that is only when there are multiple versions of a bundle are installed (with the same symbolic name):

https://docs.osgi.org/specification/osgi.core/8.0.0/framework.module.html#framework.module.fragmentbundles

Fragments can be attached to multiple hosts with the same symbolic name but different versions. If multiple fragments with the same symbolic name match the same host, then the Framework must only select one fragment, this must be the fragment with the highest version.

Ok, somehow I assume this is possible. At least the runtime didn't complain, but looking in the spec it does not look like this is permitted.

This is because Equinox just uses the first one listed:

equinox/bundles/org.eclipse.osgi/container/src/org/eclipse/osgi/container/builders/OSGiManifestBuilderFactory.java

Line 501 in 0582356

ManifestElement fragmentHost = fragmentHosts[0];

[laeubi]

@tjwatson is osgi.support.multipleHosts=true gone already?
https://www.eclipse.org/forums/index.php?t=msg&th=276829&goto=1052923&#msg_1052923

[tjwatson]

Regardless of my answer (which I don't recall off the top of my head), that was only about resolve against multiple host versions (with same BSN).

[laeubi]

I see.. then I maybe has wrong memories here...

[tjwatson]

How do we ensure what bundle is providing javax.servlet? Is that a bit random at the moment? I ask because I thought we used to use the "official" javax.servlet:javax.servlet-api one. But now, in my current installation, I see org.eclipse.jetty.servlet-api_4.0.6 is installed. I assume this is because we let p2 just find any available provider in the available repositories for the Import-Package requirement for the javax.servlet packages. For the moment that is pulling in the one from the jetty repository.

So if we move to use the org.apache.felix.http.servlet-api how do we ensure that is the ONLY one we include in the platform? Will that further complicate others that try to include the jetty one? It seems like it maybe more simple just to have our http.servlet implementation provide the contract or perhaps get the org.eclipse.jetty.servlet-api bundle to provide the contract.

[HannesWell]

How do we ensure what bundle is providing javax.servlet? Is that a bit random at the moment? I ask because I thought we used to use the "official" javax.servlet:javax.servlet-api one. But now, in my current installation, I see org.eclipse.jetty.servlet-api_4.0.6 is installed. I assume this is because we let p2 just find any available provider in the available repositories for the Import-Package requirement for the javax.servlet packages. For the moment that is pulling in the one from the jetty repository.

That's a good question, I cannot fully answer.
At the moment it is probably random, unless something strictly requires the jetty bundle as provider, but I haven't checked that.
At least if the osgi http/whiteboard bundles are in an application, they will pull in the felxi-servlet-api bundles since that's the only one that privdes the JavaServlet contract as well as the javax.servlet packages in version2.

For the record, the jetty servlet
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/ac29a0e8fd8679b4565b6ad3b5c85d3d81d4f65a/eclipse.platform.releng.prereqs.sdk/eclipse-sdk-prereqs.target#L230-L235

So if we move to use the org.apache.felix.http.servlet-api how do we ensure that is the ONLY one we include in the platform? Will that further complicate others that try to include the jetty one? It seems like it maybe more simple just to have our http.servlet implementation provide the contract or perhaps get the org.eclipse.jetty.servlet-api bundle to provide the contract.

I cannot answer that with confidence either.
Comparing both jars shows that the felxi-servlet-api is at least a real super-set of the jetty-servlet api when considering only java-classes. But there are some resource files, e.g. some translations files like LocalStrings_es.properties or

• javax\servlet\resources\permissions_7.xsd
• javax\servlet\catalog.xml

which are only present in jetty's servlet api jar.
Testwise I have updated eclipse-platform/eclipse.platform.releng.aggregator#239 to also remove jetty's servlet api bundle.
@akurtakov or @merks can you tell if org.apache.felix.http.servlet-api can replace jetty-servlet-api?

But just adding the contract is not sufficient since org.osgi.service.http imports the javax.servlet(.http) package in version 2 up to, but excluding version 3. org.osgi.service.http.whiteboard on the other hand imports both packages without a version range.

[akurtakov]

I honestly don't know. What I have always cared for is have latest jetty and have help webapp working - whatever satisfies it is good enough.

[merks]

As @akurtakov suggests, I think we can only test that help works. Generally I test that by launching runtime workspace and seeing in Help -> Contents works, checking which bundles are installed...

[tjwatson]

But just adding the contract is not sufficient since org.osgi.service.http imports the javax.servlet(.http) package in version 2 up to, but excluding version 3. org.osgi.service.http.whiteboard on the other hand imports both packages without a version range.

This implies we need a single bundle exporting two versions of the same package. While possible, it is not truthful. I am not sure I agree that we should move in that direction. To that end, I really think we should repackage the deprecated org.osgi.service.http classes into our own bundle that imports the servlet packages at the right version we need. That is the correct way to do this.

[tjwatson]

To that end, I really think we should repackage the deprecated org.osgi.service.http classes into our own bundle that imports the servlet packages at the right version we need. That is the correct way to do this.

Actually, we should import with no version and do the contract dependency like the OSGi provided org.osgi.service.http.whiteboard bundle does.

[laeubi]

I also want to note that it seems the org.osgi.service.http is already pulled in I get a lot of warnings in the maven build like this:

Problems resolving provisioning plan.: [Unable to satisfy dependency from org.osgi.service.http 1.2.2.202109301733 to java.package; javax.servlet [2.1.0,3.0.0).; Unable to satisfy dependency from org.osgi.service.http 1.2.2.202109301733 to java.package; javax.servlet.http [2.1.0,3.0.0).; Unable to satisfy dependency from org.osgi.service.http.whiteboard 1.1.1.202109301733 to osgi.contract; (&(osgi.contract=JavaServlet)(version=3.1.0)).]

what is quite annoying it would be great to get a solution for this as this currently hinder us from using the Quality gate as this is reported as new warnings on maven problem parser.

[HannesWell]

To that end, I really think we should repackage the deprecated org.osgi.service.http classes into our own bundle that imports the servlet packages at the right version we need. That is the correct way to do this.

Actually, we should import with no version and do the contract dependency like the OSGi provided org.osgi.service.http.whiteboard bundle does.

Should I just place the API sources into the Implementation bundle and export+import them or create a separate bundle?

[HannesWell]

Thanks Ed for the analysis

The requirements on the exported packages will automatically vanish respectively will be redirected, once this PR is completed and the http packages are not contained in this Plugin/Bundle anymore.
The requirements on this the o.e.osgi.service bundle should be replaced by Imports of the required packages. Then the requirements are also automatically 're-directed' to the official osgi jars, too.

Correspondingly the inclusion in features should just be removed. I would only keep it in at least one equinox feature so that it is still included in the sdk repo and in the TP of consumers that still use it.

I'll reach out to the affected projects ones this is completed.

I guess it will be quite some significant period of time before there are no requirements on the bundle anymore....

I assumed that we have the usual two years deprecation period?

[HannesWell]

@tjwatson could you please share your assessment of the approach to use a fragment for javax/jakarta.servlet-api, that is currently implemented by this PR?

Outdated

[tjwatson]

@tjwatson could you please share your assessment of the approach to use a fragment for javax/jakarta.servlet-api, that is currently implemented by this PR?

I'll refer back to my comment #403 (comment)

This implies we need a single bundle exporting two versions of the same package. While possible, it is not truthful. I am not sure I agree that we should move in that direction. To that end, I really think we should repackage the deprecated org.osgi.service.http classes into our own bundle that imports the servlet packages at the right version we need. That is the correct way to do this.

I still stand by this approach and I will refer to my previous comment #403 (comment)

We need to stop trying to think of ways to use an old deprecated bundle binary from another project. We still need the API, we should maintain the API ourselves since the OSGi WG has given up on it. Once we stop using the API altogether then we can remove it.

I've not seen any arguments against this approach that involves no hacking, n repackaging, and no confusing multi-version exports.

[HannesWell]

We need to stop trying to think of ways to use an old deprecated bundle binary from another project. We still need the API, we should maintain the API ourselves since the OSGi WG has given up on it. Once we stop using the API altogether then we can remove it.

I've not seen any arguments against this approach that involves no hacking, n repackaging, and no confusing multi-version exports.

OK, I just wanted to make sure that this suggestion is not lost because I had the impression that it was confused with a fragment for o.o.service.http instead of jakarta/javax.servlet.
Then lets move the org.osgi.service.http classes into a new org.eclipse.equinox.http.service.api bundle. When all elements that reference something from org.osgi.service.http we can just deprecate and eventually remove the bundle with them.

I have update this PR to move the classes into a new org.eclipse.equinox.http.service.api bundle. At the moment the project is configured to copy the sources in the bundle using the maven-depencency-plugin on the fly during CI and workspace builds.
If you prefer to have the sources checked into git, I can change that as well.
Are you then fine with this PR as it is?

[tjwatson]

Are you then fine with this PR as it is?

How does that work with the projects imported into the IDE? Does PDE and M2E handle this and get everything compiled correctly in the workspace?

[HannesWell]

Are you then fine with this PR as it is?

How does that work with the projects imported into the IDE? Does PDE and M2E handle this and get everything compiled correctly in the workspace?

Yes. Of course m2e has to be installed. But IIRC it is already part of the Oomph development setup for the SDK.

[tjwatson]

Yes. Of course m2e has to be installed. But IIRC it is already part of the Oomph development setup for the SDK.

Then I am fine with this approach, but if there are any issues we should simply place the .java source into the project and be done with it.

[merks]

I'm such a non-fan to not checking in code, but that's just my opinion. Please be 100% sure that the Oomph setup actually works out-of-the box from a freshly cloned clone.

[tjwatson]

I'm such a non-fan to not checking in code, but that's just my opinion.

Not just yours, that is mine also. It always seems more simple to just have the source checked into the source repo vs. trying to pull it out of some pre-existing artifact.

[tjwatson]

trying to pull it out of some pre-existing artifact.

One advantage I can think of is that it would stop others from opening PRs trying to "correct" the formatting of the source one way or another :)

[HannesWell]

Yes. Of course m2e has to be installed. But IIRC it is already part of the Oomph development setup for the SDK.

Then I am fine with this approach,

Great. I'm glad we can finally complete this. Thank you all for the discussion and your patience.

I'm such a non-fan to not checking in code, but that's just my opinion. Please be 100% sure that the Oomph setup actually works out-of-the box from a freshly cloned clone.

I verified that and for me it worked well to import this and fetch the sources just with the Oomph setup. I'm never 100% sure about anything but I'm very confident it works if m2e is installed. :)
But because I have the impression that, contrary to my previous assumption m2e is not installed with the SDK setup and have now kept the sources checked-in in the SCM.

trying to pull it out of some pre-existing artifact.

One advantage I can think of is that it would stop others from opening PRs trying to "correct" the formatting of the source one way or another :)

That's right. And it is also simpler to upgrade to a new version because only the version number in the pom (and the manifest in Manifest-First plugins) needs to be updated. For this abandoned spec that's not so interesting, but for example for the embedded sources in org.eclipse.osgi that might be of interest.
If you are interested and we agree to make m2e mandatory for SDK development I can provide a PR to implement that.

[tjwatson]

If you are interested and we agree to make m2e mandatory for SDK development I can provide a PR to implement that.

I would rather not. There are cases where we modify the internals of the org.osgi source in that project. I believe it is more simple to keep the source as is. Besides that, being used as the implementation during the development of the specification there are times where we need the source in our repo before OSGi has even published a source artifact at all.