fatal error: concurrent map read and map write

[czarcas7ic]

Background

When creating a block explorer on a archive node, pruning nothing, with big dipper v2 (using bdjuno), the following error occurs at what seems like random intervals:

https://gist.github.com/rkben/8814376162cda8c08dd33d60f6933429#file-cleand_osmo-log-L256

I instructed mp20 to remove the snapshot interval in case bdjuno reading the db files while the snapshot being created is causing this issue

Acceptance Criteria

• Investigate the problem
• create a pr with a potential fix
• create a test tag and distribute to the user who reported the problem

[p0mvn]

@czarcas7ic thanks for creating it

From the initial investigation, there seems to be a concurrency issue with accessing a common resource inside nodedb.

https://gist.github.com/rkben/8814376162cda8c08dd33d60f6933429#file-cleand_osmo-log-L1257 - this goroutine seems to be committing a block
https://gist.github.com/rkben/8814376162cda8c08dd33d60f6933429#file-cleand_osmo-log-L268 - this goroutine seems to be handling the grpc query

They both access the same resource and fatal error. Investigating further...

[p0mvn]

This should be merged now, closing

[p0mvn]

The solution deadlocked during testing, reopening

[p0mvn]

The solution is now released under 7.0.4. Waiting for feedback from validators before closing this.

[p0mvn]

The fix is working even with the concurrent gateway, closing this issue for now