p2 director does not handle GOAWAY gracefully

[sithmein]

We recently updated to Eclipse 2024-03 and now installation of a larger set of features through the p2 director (on the command line) fail with java.io.IOException: /192.168.187.2:42772: GOAWAY received. It seems with this version HTTP/2 is used and our repository server (nginx) has a limit of 1,000 requests per connection. Hence it sends GOAWAY but apparently the p2 director does not handle this (quite common) case. Here is a complete stacktrace:

08:22:43.838  !MESSAGE Provisioning exception
08:22:43.838  !STACK 1
08:22:43.838  org.eclipse.equinox.p2.core.ProvisionException: Unable to read repository at https://xxx/yyy/artifacts.xml.xz.
08:22:43.838  	at org.eclipse.equinox.internal.p2.repository.CacheManager.createCacheFromFile(CacheManager.java:132)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.simple.XZedSimpleArtifactRepositoryFactory.getLocalFile(XZedSimpleArtifactRepositoryFactory.java:65)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.simple.XZedSimpleArtifactRepositoryFactory.load(XZedSimpleArtifactRepositoryFactory.java:89)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.simple.XZedSimpleArtifactRepositoryFactory.load(XZedSimpleArtifactRepositoryFactory.java:42)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.ArtifactRepositoryManager.factoryLoad(ArtifactRepositoryManager.java:92)
08:22:43.838  	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:791)
08:22:43.838  	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:689)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.ArtifactRepositoryManager.loadRepository(ArtifactRepositoryManager.java:131)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.ArtifactRepositoryManager.loadRepository(ArtifactRepositoryManager.java:125)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepository.load(CompositeArtifactRepository.java:476)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepository.addChild(CompositeArtifactRepository.java:157)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepository.<init>(CompositeArtifactRepository.java:79)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepositoryFactory.load(CompositeArtifactRepositoryFactory.java:120)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.ArtifactRepositoryManager.factoryLoad(ArtifactRepositoryManager.java:92)
08:22:43.838  	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:791)
08:22:43.838  	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:689)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.ArtifactRepositoryManager.loadRepository(ArtifactRepositoryManager.java:131)
08:22:43.838  	at org.eclipse.equinox.internal.p2.artifact.repository.ArtifactRepositoryManager.loadRepository(ArtifactRepositoryManager.java:125)
08:22:43.838  	at org.eclipse.equinox.internal.p2.director.app.DirectorApplication.initializeRepositories(DirectorApplication.java:807)
08:22:43.839  	at org.eclipse.equinox.internal.p2.director.app.DirectorApplication.run(DirectorApplication.java:1429)
08:22:43.839  	at org.eclipse.equinox.internal.p2.director.app.DirectorApplication.start(DirectorApplication.java:1622)
08:22:43.839  	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:208)
08:22:43.839  	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:143)
08:22:43.839  	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:109)
08:22:43.839  	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:439)
08:22:43.839  	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:271)
08:22:43.839  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
08:22:43.839  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
08:22:43.839  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
08:22:43.839  	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
08:22:43.839  	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:651)
08:22:43.839  	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:588)
08:22:43.839  	at org.eclipse.equinox.launcher.Main.run(Main.java:1459)
08:22:43.839  	at org.eclipse.equinox.launcher.Main.main(Main.java:1432)
08:22:43.839  Caused by: java.io.IOException: /192.168.187.2:42772: GOAWAY received
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection.handleGoAway(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection.handleConnectionFrame(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection.processFrame(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.frame.FramesDecoder.decode(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection$FramesController.processReceivedData(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection.asyncReceive(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.processQueue(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.runOrSchedule(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.Http2Connection$Http2TubeSubscriber.onNext(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SSLTube$DelegateWrapper.onNext(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onNext(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(Unknown Source)
08:22:43.839  	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(Unknown Source)
08:22:43.839  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
08:22:43.839  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
08:22:43.839  	at java.base/java.lang.Thread.run(Unknown Source)

[merks]

I believe this issue only arises if the org.eclipse.ecf.provider.filetransfer.httpclientjava provider is used. You can use this system system property to disable this provided and as a result use the older provider that uses Apache libraries:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava

Please 🙏 let us know if this works for you.

[laeubi]

thanks for the report, beside the (temporal) solution from @merks I thin we should enhance the cache manager here to retry such attempts, if I remember correctly there are already some handling for specific cases.

[sithmein]

I tried the system property. The requests are now HTTP/1.1 (which means there is no GOAWAY anyway). However, there is a different problem now:

!MESSAGE No repository found at https://xxx/yyy/master%2creleases*2024-06%2c.

No further information available right now. Could the %2c at the end be the problem? It's not a problem without the system property.

[merks]

I can't really say where such a URI would come from. I know there have been changes in the Apache libraries used by the older provider too. What URI would actually work in the Repository Explorer in the IDE; all the Eclipse IDEs also suppress the new providers so try testing your URIs in the IDE with the Repository Explorer...

[sithmein]

The %2c and * come from our system, but they are valid characters in a URI and calling that URL with other tools (curl) or even older versions of the p2 director worked fine.

[laeubi]

@sithmein it is completely impossible to reliable guess whats wrong with a site just given some random characters.

So please provide us with a public accessible site that reproduces the problem it does not need to contain the same content like your site of course.

[sithmein]

Try https://update-origin.knime.com/analytics-platform/tmp/master%2creleases*2024-06%2c as a p2 repository URL.
https://update-origin.knime.com/analytics-platform/tmp/master%2creleases*2024-06%2c/compositeContent.jar (and compositeArtifacts.jar) can be downloaded from the browser but if the above URL is used in Eclipse 2024-03 it cannot find a repository.

[merks]

Strange, the Repository Explorer loads but you can't install from it:

When I convert the %2c to to , then it works:

I'll try to find time to debug what's going wrong...

[merks]

The call stack calls this utility method that clearly states the string must be one that is not encoded and even mentiones the double encoding of % if that's present:

So indeed, by the time we get to creating a URI, it's encoded the %25.

I expect if we were to change this, we'd break someone else who's relying on their text getting encoded so I don't see a good way to fix this.

Using this directly works:

https://update-origin.knime.com/analytics-platform/tmp/master,releases*2024-06,/

Isn't that sufficient?

[sithmein]

In this instance probably yes. However , is a reserved character in a URI and therefore has to be encoded if used as a "normal" character.
I know that this is a rather special case but the problem indicates that something is fishy with how URIs are handled.

[merks]

I suppose yes, but Java happily creates a URI with a ,:

And the Repository Explorer which does no encoding also works fine with a ,:

In any case, it's the encoding of an already-encoded string that's a problem and I see no good way to avoid it without the risk of breaking someone else...