Datasource connection deadlock #3173
Comments
|
This is the log of ebean getting jdbc connection that I debugged, uuid is the id of the connection, you can see from the log that when the connection 89dd1f90-72cf-47d8-93a6-990e1ec48bc0 is not closed, it gets another connection 827c9140-2345-4d3e-8220-f0c5bf1bba73, this problem causes deadlock when getting connection in high concurrency and the connection pool limits the maximum number. |
|
ebean version 12.8.3 |
So why do you want hashCode() to trigger lazy loading? Surely you don't want that to happen? 1. Change from using The recommendation with Ebean is to use With Ebean we promote the use of List and in fact in version 15.x we might even enforce use of List (and even remove support for Set). 2. Remove the You could for example remove So unless you really know what you are doing remove the |
|
But overriding hashcode is a very common requirement in Entity, for example, in this case, we need to use a Set collection to deduplicate a batch of objects externally, why can’t we reuse the connection through the context?
@Column(name = "reference")
private final String reference;
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((reference == null) ? 0 : reference.hashCode());
return result;
} |
|
This The "Effective Java" recommendation is that
Most use cases are better supported by findMap(). In this case perhaps the map key is It isn't clear to me yet what you are trying to achieve but this hashCode() implementation is going to be a problem for you.
Not sure. Can't really see enough of the code yet to follow what is being done here. |
|
Why does hashcode change when reference is defined as immutable? |
I believe there is no explicit transaction created. Ebean is implicitly creating a transaction as needed to run the query. So in So I think this is why the connection is not being re-used when later on the hashCode invokes lazy loading. And then that leads to obtaining another connection from the dataSource pool etc. So workarounds, assuming you want to keep the hashCode implementation could be:
|
So yes, this code indeed looks to be ok because its a final field - my bad, I missed that. So yes the value for reference should not mutate so the hashCode() should always return the same value. |
Looking at the code, it does like a bug here in that ... In OrmQueryRequest we have Looking at Looking at So looking at this as a bug, we'd look at a change such that the "read query" is ALSO put into transaction scope in initTransIfRequired() and also cleared from transaction scope at the end via endTransIfRequired(). With that change in place, then any lazy loading invoked internally while executing a "read only" query with implicit transactions would reuse the transaction/java.sql.Connection [and hence not get into requesting another connection from the pool]. |
Yes, I solved the problem by explicitly creating a transaction, but I only solved this one. I wonder if I can solve this problem directly at the bottom layer, instead of implicitly discovering this situation and then explicitly adding a transaction. |
|
I think we posted at the same time there.
Yes, it looks like we can fix it in |
Will this problem be solved at the bottom layer? If so, is there an approximate time? |
|
Hi Rob, I'm wondering if this issue will be resolved anytime soon? Our systems heavily depends on it and if not, it will take us a lot of efforts to put workarounds in place almost everywhere. Thanks a lot. |
|
First step is a failing test. Are you able to create a test case perhaps based on example-minimal ? https://github.com/ebean-orm-examples/example-minimal |
I have completed the test code but I don’t have permission to push it up. What should I do? |
Did you "create a fork"? You won't have permission to push to the repo directly but if you have your own fork you should be able to push to that. |
Thank you, I have submitted the test code to this address: https://github.com/lbsoft-lwsoft/example-minimal/blob/test-case-3173/src/test/java/org/example/domain/ConnectionLeakTest.java |
- The error reported was that a DataSource connection pool maxed out - The symptom was that there was an additional lazy loading queries invoked via the hashCode/equals implementation of ManyToMany Set. - The fix is for BeanSet init() to lazy load with onlyIds = false, that avoids the extra lazy loading query from being executed
#3173 - BeanSet lazy loading / Too many DataSource connections used
This shows the extra lazy loading that is occurring when clear() is called on the [Bean]Set DUE to the hashcode/equals implementation that is present on the entity bean that is in the Set.
Follow up #3173 - Add test for BeanSet clear() when hashCode/equals used
To avoid the 1+N lazy loading queries that are invoked due to the clear() when the entity bean in the Set has a hashCode/equals implementation
Follow up #3173 - Change BeanSet clear() to lazy load ALL properties
|
Hi Rob, Can this fix be updated to 12.8, we are using 12.8.3 and can’t update to 13 for now? thanks |
- BeanSet init() and initClear() load with onlyIds false because the expectation is that with BeanSet the equals/hashCode implementation can use a property - BeanMap lazy loading, include the mapKey if defined in the lazy loading query
…t-loading [12x] Backport of fix for #3173 BeanSet init(), initClear() and BeanMap
|
ebean version Note: The branch |
Expected behavior
It is hoped that the new query caused by the previous query process should use the same jdbc connection.
Actual behavior
When the previous jdbc connection is not closed, hashcode triggers lazy loading and re-acquires a new jdbc connection, which may cause mutual exclusion of connections and deadlock in concurrent situations. For example, the maximum available connection pool is 2, when two threads trigger lazy loading at the same time, because the previous two connections are not released and get new connections through the connection pool, the connection pool finds that there is no available connection and suspends the thread to wait for other threads to release the connection, and both threads are stuck in getting new connections, resulting in deadlock.
Steps to reproduce
The text was updated successfully, but these errors were encountered: