Infinispan can't be manually started with Quarkus 1.3.0 #7910

burmanm · 2020-03-17T13:21:02Z

Describe the bug
When launching the application that uses Infinispan directly (not through infinispan-embedded), the startup fails to loading metrics:

2020-03-17 13:35:10,563 ERROR [io.qua.application] (main) Failed to start application: java.lang.IllegalStateException: Display name differs from previous usage
	at io.smallrye.metrics.MetricsRegistryImpl.verifyMetadataEquality(MetricsRegistryImpl.java:203)
	at io.smallrye.metrics.MetricsRegistryImpl.register(MetricsRegistryImpl.java:151)
	at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.garbageCollectionMetrics(SmallRyeMetricsRecorder.java:250)
	at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.registerBaseMetrics(SmallRyeMetricsRecorder.java:110)
	at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics18.deploy_0(SmallRyeMetricsProcessor$registerBaseAndVendorMetrics18.zig:65)
	at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics18.deploy(SmallRyeMetricsProcessor$registerBaseAndVendorMetrics18.zig:36)
	at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:138)
	at io.quarkus.runtime.Application.start(Application.java:90)
	at io.quarkus.runtime.Application.run(Application.java:228)
	at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)

This is visible here because of #7907 that causes the displayName to differ from the one that Infinispan loaded, since Infinispan's new DefaultCacheManager(is) loads its own metrics before Quarkus:

DefaultMetadata{name='gc.total', type=counter, unit='none', reusable=true, description='Displays the total number of collections that have occurred. This attribute lists -1 if the collection count is undefined for this collector.', displayName='Garbage Collection Count'}
 
register:116, MetricsRegistryImpl (io.smallrye.metrics)
register:74, JmxRegistrar (io.smallrye.metrics.setup)
register:58, JmxRegistrar (io.smallrye.metrics.setup)
init:50, JmxRegistrar (io.smallrye.metrics.setup)
start:66, ApplicationMetricsRegistry (org.infinispan.metrics.impl)
start:41, CorePackageImpl$2 (org.infinispan.metrics.impl)
start:39, CorePackageImpl$2 (org.infinispan.metrics.impl)
invokeStart:587, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:578, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
startDependencies:605, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:569, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
startDependencies:605, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:569, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
startDependencies:605, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:569, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
registerComponentInternal:121, AbstractComponentRegistry (org.infinispan.factories)
registerComponent:110, AbstractComponentRegistry (org.infinispan.factories)
registerComponent:102, AbstractComponentRegistry (org.infinispan.factories)
cacheManagerStarting:106, LifecycleManager (org.infinispan.query.core.impl)
modulesManagerStarting:271, GlobalComponentRegistry (org.infinispan.factories)
access$000:65, GlobalComponentRegistry (org.infinispan.factories)
start:361, GlobalComponentRegistry$ModuleInitializer (org.infinispan.factories)
start:173, CorePackageImpl$11 (org.infinispan.factories)
start:171, CorePackageImpl$11 (org.infinispan.factories)
invokeStart:587, BasicComponentRegistryImpl (org.infinispan.factories.impl)
doStartWrapper:578, BasicComponentRegistryImpl (org.infinispan.factories.impl)
startWrapper:547, BasicComponentRegistryImpl (org.infinispan.factories.impl)
access$700:30, BasicComponentRegistryImpl (org.infinispan.factories.impl)
running:770, BasicComponentRegistryImpl$ComponentWrapper (org.infinispan.factories.impl)
preStart:254, GlobalComponentRegistry (org.infinispan.factories)
start:238, AbstractComponentRegistry (org.infinispan.factories)
internalStart:742, DefaultCacheManager (org.infinispan.manager)
start:713, DefaultCacheManager (org.infinispan.manager)
<init>:391, DefaultCacheManager (org.infinispan.manager)
<init>:335, DefaultCacheManager (org.infinispan.manager)
<init>:322, DefaultCacheManager (org.infinispan.manager)
init:64, IspnCacheManager (org.hawkular.alerts.cache)

And there the displayName is correct, while with Quarkus it isn't. I'm not sure what changed between 1.2.1.Final and 1.3.0.Final (Infinispan 10.0.0 -> 10.1.2.Final), but now the metrics are loaded multiple times and that breaks things.

Expected behavior
Infinispan would continue working as it did before.

Actual behavior
See above.

To Reproduce
Steps to reproduce the behavior:

https://github.com/burmanm/custom-policies-engine/tree/infinispan_changes
@singleton's static { } blocks loads: cacheManager = new DefaultCacheManager(is);
Kaboom?

Configuration

Screenshots

Additional context

The text was updated successfully, but these errors were encountered:

geoand · 2020-03-17T13:23:17Z

cc @jmartisk @karesti @wburns

jmartisk · 2020-03-17T13:29:37Z

This is the Infinispan commit after which it broke: infinispan/infinispan@3dad7af#diff-06d9b67f22d98c28952c53d825f1d4f3R62-R74
Infinispan now starts registering base metrics by itself even though only Quarkus should be doing that. I think that Infinispan needs a new option to turn base metrics off.

geoand · 2020-03-17T13:30:54Z

@jmartisk what would be the effect of fixing #7907 in Quarkus with regards to this issue?

jmartisk · 2020-03-17T13:32:54Z

@geoand I think (didn't check) that then the metrics would just be registered twice, and the second registrations would overwrite the first ones, so while that would be unnecessary, in practice it would work correctly even without fixing the Infinispan part

geoand · 2020-03-17T13:35:04Z

That sounds a whole lot better than not being able to start the application at all :)

jmartisk · 2020-03-17T13:49:51Z

Hmm, so in the end, no it actually wouldn't work because there are Gauge metrics, and Gauges can't be re-registered without removing the original Gauge first... In that case I'm getting a different exception

Caused by: java.lang.IllegalArgumentException: A metric with metricID MetricID{name='classloader.loadedClasses.count', tags=[]} already exists
	at io.smallrye.metrics.MetricsRegistryImpl.register(MetricsRegistryImpl.java:131)
	at io.smallrye.metrics.MetricsRegistryImpl.register(MetricsRegistryImpl.java:111)
	at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.classLoadingMetrics(SmallRyeMetricsRecorder.java:282)
	at io.quarkus.smallrye.metrics.runtime.SmallRyeMetricsRecorder.registerBaseMetrics(SmallRyeMetricsRecorder.java:111)
	at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics7.deploy_0(SmallRyeMetricsProcessor$registerBaseAndVendorMetrics7.zig:65)
	at io.quarkus.deployment.steps.SmallRyeMetricsProcessor$registerBaseAndVendorMetrics7.deploy(SmallRyeMetricsProcessor$registerBaseAndVendorMetrics7.zig:36)
	at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:106)
	... 13 more

geoand · 2020-03-17T13:50:45Z

OK, so we really need an Infinispan release to fix this issue I guess...

We'll also need some tests to make sure this doesn't happen again

gsmet · 2020-03-17T14:06:31Z

@jmartisk so we could unregister the gauges added by Infinispan to add our own? Are we sure ours will always be registered after the ones from Infinispan?

Asking that because I think we need a contingency plan in case we can't get an Infinispan release by the end of the month.

gsmet · 2020-03-17T14:06:58Z

Also maybe you should open an Infinispan issue describing the problem?

gsmet · 2020-03-17T14:07:41Z

And another option: if Infinispan now has its own metrics, maybe we should just remove the Quarkus one and let Infinispan do what they think is best for them?

burmanm · 2020-03-17T14:14:25Z

@gsmet That probably won't work, since Infinispan registers the GC, JVM, etc metrics. I don't think Quarkus users without Infinispan want to live without those metrics.

gsmet · 2020-03-17T15:09:01Z

Argh.

/cc @karesti @wburns
/cc @Sanne too while I'm at it

anistor · 2020-03-17T15:23:37Z

This is the Infinispan commit after which it broke: infinispan/infinispan@3dad7af#diff-06d9b67f22d98c28952c53d825f1d4f3R62-R74
Infinispan now starts registering base metrics by itself even though only Quarkus should be doing that. I think that Infinispan needs a new option to turn base metrics off.

This is something wrong that we noticed also and is already fixed in the newly released 10.1.5.Final

anistor · 2020-03-17T15:24:53Z

@gsmet That probably won't work, since Infinispan registers the GC, JVM, etc metrics. I don't think Quarkus users without Infinispan want to live without those metrics.

Infinispan embedded not longer does that in 10.1.5.final. Only the infinispan server continues to register those base metrics.

wburns · 2020-03-17T15:39:20Z

@gsmet I can try to get a PR with the updated version, but have to get my quarkus env working again :) Should have one soon hopefully, and assuming it passes tests.

wburns · 2020-03-17T16:37:15Z

I created #7921 which fixes the client, but I just ran embedded and need to update some xml test files it appears.

jmartisk · 2020-03-18T07:04:47Z

I ran some experiments with #7921 included and I no longer can reproduce the problem. Perhaps @burmanm could you check it too and then we can close this issue?

Sanne · 2020-03-18T09:45:56Z

I'm confused about the premise:

the application that uses Infinispan directly (not through infinispan-embedded)

is this a supported configuration? Not as far as I know: one should either use the infinispan-embedded extension, or use the server. Incidentally, the infinispan-embedded extension was created by wburns exclusively to make Infinispan Server experiments with Quarkus.

Could you tell us more about your use case please?

gsmet · 2020-03-18T10:04:01Z

@Sanne I think he means he uses the Infinispan client extension and not the embedded one.

Sanne · 2020-03-18T10:13:01Z

That doesn't seem to be the case.

anistor · 2020-03-18T10:21:08Z

The ispn client does not expose MP metrics yet. That can't be the issue.

burmanm · 2020-03-18T10:50:23Z

@Sanne We create our own EmbeddedCacheManager in a different maven artifact than the one that runs Quarkus (with some logic which defines which configuration file is loaded).

This module is then embedded to our Quarkus service, which doesn't know about the Infinispan at all - what it sees is the interfaces to this module which may or may not use Infinispan (offtopic: and might indeed use a mixture of Infinispan and other services to manage the data, although I'm hoping to use Infinispan as a pretty RocksDB wrapper for most use-cases).

The other module has no knowledge of Quarkus and it's also used outside Quarkus. It makes a spaghetti code if I have to instantiate my Infinispan inside Quarkus only to start the other module by giving it the cacheManager (thus exposing internal implementation) and then do the same spaghetti code for every other runtime as well. That would pretty much be the end of any good OO principle.

burmanm · 2020-03-18T13:02:34Z

@jmartisk Yep, works with the Quarkus snapshot after #7921

Sanne · 2020-03-18T14:32:06Z

@burmanm you're bringing up some interesting points, we should probably write up some more about how we expect such designs to evolve; but I won't presume to write any general recommendations yet as many of these are novel ideas: for proper guidance to emerge we'll need more feedback like these and see what works best.

To put things into perspective: while this specific problem might have been solved already by #7921 , it's probably best you keep in mind that you're doing something we don't support and don't have integration tests for, so it's possible it will break again without notice. Let me try to explain why and how we expect designs to change - but again I won't presume we have thought of all aspects, so hopefully you'll be able to help figuring this all out.

In Quarkus it's fine to use 3rd party libraries which don't have an extension, but in this case:

[obviously] we don't test for them
we can't help you making them work in GraalVM native image
you won't benefit from Live Reload capabilities, e.g. if the Infinispan configuration file is changed Quarkus won't reload
the specific component will not benefit from our optimisations, so possibly booting slower and consuming more runtime memory

For most "normal" Java code that people typically develop in house this is not a problem; an extension is only going to be needed for highly dynamic code, complex frameworks, libraries using bytecode enhancement, instrumentation, etc..

So while encapsulation is of course a highly valuable concept, I don't think we can compare instantiation of a complex service like an in-memory data grid cluster component with a straight-forward encapsulation need: such a component needs tight integration with the platform so its initialization, but also integration with management, metrics, operations need to "tie in" into the core platform. Typically one uses a service lookup for such aspects, and you're expected to have your own custom code "lookup" some service which is started and managed by the platform.

Wouldn't it be better to use the dependency injection mechanism? Quarkus offers great decoupling of services via CDI - in fact most of our own intregration is relying on it.

Offload the initialization and management details of such a component to the platform, and have your component simply inject a reference to the caches. Your other non-Quarkus services can easily provide a simimlar factory, thath's hardly figthing good OO principles.

Wouldn't that be better?

burmanm added the kind/bug Something isn't working label Mar 17, 2020

geoand added the area/infinispan Infinispan label Mar 17, 2020

wburns mentioned this issue Mar 17, 2020

Update Infinispan to 10.1.5.Final and update test classes #7921

Merged

gsmet added the triage/backport? label Mar 18, 2020

wburns mentioned this issue Mar 18, 2020

Add SmallRye Metrics extension to Infinispan Embedded #7935

Merged

gsmet removed the triage/backport? label Mar 21, 2020

gsmet added this to the 1.3.1.Final milestone Mar 21, 2020

gsmet closed this as completed Mar 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infinispan can't be manually started with Quarkus 1.3.0 #7910

Infinispan can't be manually started with Quarkus 1.3.0 #7910

burmanm commented Mar 17, 2020

geoand commented Mar 17, 2020 •

edited

Loading

jmartisk commented Mar 17, 2020

geoand commented Mar 17, 2020

jmartisk commented Mar 17, 2020 •

edited

Loading

geoand commented Mar 17, 2020

jmartisk commented Mar 17, 2020

geoand commented Mar 17, 2020 •

edited

Loading

gsmet commented Mar 17, 2020

gsmet commented Mar 17, 2020

gsmet commented Mar 17, 2020

burmanm commented Mar 17, 2020

gsmet commented Mar 17, 2020

anistor commented Mar 17, 2020

anistor commented Mar 17, 2020

wburns commented Mar 17, 2020

wburns commented Mar 17, 2020

jmartisk commented Mar 18, 2020

Sanne commented Mar 18, 2020

gsmet commented Mar 18, 2020

Sanne commented Mar 18, 2020

anistor commented Mar 18, 2020

burmanm commented Mar 18, 2020

burmanm commented Mar 18, 2020

Sanne commented Mar 18, 2020 •

edited

Loading

Infinispan can't be manually started with Quarkus 1.3.0 #7910

Infinispan can't be manually started with Quarkus 1.3.0 #7910

Comments

burmanm commented Mar 17, 2020

geoand commented Mar 17, 2020 • edited Loading

jmartisk commented Mar 17, 2020

geoand commented Mar 17, 2020

jmartisk commented Mar 17, 2020 • edited Loading

geoand commented Mar 17, 2020

jmartisk commented Mar 17, 2020

geoand commented Mar 17, 2020 • edited Loading

gsmet commented Mar 17, 2020

gsmet commented Mar 17, 2020

gsmet commented Mar 17, 2020

burmanm commented Mar 17, 2020

gsmet commented Mar 17, 2020

anistor commented Mar 17, 2020

anistor commented Mar 17, 2020

wburns commented Mar 17, 2020

wburns commented Mar 17, 2020

jmartisk commented Mar 18, 2020

Sanne commented Mar 18, 2020

gsmet commented Mar 18, 2020

Sanne commented Mar 18, 2020

anistor commented Mar 18, 2020

burmanm commented Mar 18, 2020

burmanm commented Mar 18, 2020

Sanne commented Mar 18, 2020 • edited Loading

geoand commented Mar 17, 2020 •

edited

Loading

jmartisk commented Mar 17, 2020 •

edited

Loading

geoand commented Mar 17, 2020 •

edited

Loading

Sanne commented Mar 18, 2020 •

edited

Loading