fix(gatsby): use lmdb.removeSync so getNode can't return deleted nodes #33554
Conversation
gatsbot
bot
added
the
status: triage needed
KyleAMathews
added
type: bug
KyleAMathews
changed the title
|
Hmm, something is not right with this PR - it includes commits that were already merged to master? |
…Node can still return deleted node
|
@vladar fixed :) |
| .map(nodeId => { | ||
| if (preSyncDeletedNodeIdsCache.has(nodeId)) { | ||
| return undefined | ||
| } | ||
|
|
||
| return getNode(nodeId) | ||
| }) |
There was a problem hiding this comment.
This seems redundant as getNode has the same check now.
| .map(nodeId => { | |
| if (preSyncDeletedNodeIdsCache.has(nodeId)) { | |
| return undefined | |
| } | |
| return getNode(nodeId) | |
| }) | |
| .map(nodeId =>getNode(nodeId)!) |
| const stats = getDatabases().nodes.getStats() | ||
| // @ts-ignore | ||
| return Number(stats.entryCount || 0) // FIXME: add -1 when restoring shared structures key | ||
| const stats = getDatabases().nodes.getStats() as { entryCount: number } |
| lastOperationPromise.then(() => { | ||
| preSyncDeletedNodeIdsCache.clear() | ||
| }) |
There was a problem hiding this comment.
I think this can lead to a race condition when some DELETE_NODE is called while the transaction is being submitted by lmdb in another thread. Ideally, if clear is pending and DELETE_NODE happens we should cancel the previous clear attempt.
Pretty edge case, but it will be very hard to debug if we actually hit it.
|
|
||
| if (action.type === `DELETE_NODE`) { | ||
| preSyncDeletedNodeIdsCache.add( | ||
| ((action as IDeleteNodeAction).payload as IGatsbyNode).id |
There was a problem hiding this comment.
https://app.circleci.com/pipelines/github/gatsbyjs/gatsby/76288/workflows/e843318b-e1c5-4cdf-b155-b82bf562e44e/jobs/898140 shows cases where DELETE_NODE payload might be falsy:
error Cannot read property 'id' of undefined
TypeError: Cannot read property 'id' of undefined
- lmdb-datastore.ts:218 updateDataStore
[artifacts]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:218:66
- lmdb-datastore.ts:259 callback
[artifacts]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:259:7
- mett.ts:50 forEach
[artifacts]/[gatsby]/src/utils/mett.ts:50:11
- Set.forEach
- mett.ts:49 Object.emit
[artifacts]/[gatsby]/src/utils/mett.ts:49:17
- index.ts:153
[artifacts]/[gatsby]/src/redux/index.ts:153:11
so we should check if payload is truthy before reaching to id. We do checks like that in other places handling DELETE_NODE action - for example in
gatsby/packages/gatsby/src/datastore/lmdb/updates/nodes.ts
Lines 16 to 21 in 0f421db
There was a problem hiding this comment.
also casting payload as IGatsbyNode pretty much removes information that this might be falsy as our action type does mention it:
gatsby/packages/gatsby/src/redux/types.ts
Lines 836 to 840 in 0f421db
I really think we should avoid casts like this. In fact none of casts in this line are needed because after checking for type action already become that specific type in closure below
#33554) Co-authored-by: Ward Peeters <[email protected]> (cherry picked from commit 98a843c) # Conflicts: # packages/gatsby/src/datastore/lmdb/lmdb-datastore.ts
#33554) (#33633) Co-authored-by: Ward Peeters <[email protected]> (cherry picked from commit 98a843c) # Conflicts: # packages/gatsby/src/datastore/lmdb/lmdb-datastore.ts Co-authored-by: Kyle Mathews <[email protected]>
gatsbyjs#33554) Co-authored-by: Ward Peeters <[email protected]>
#33554) Co-authored-by: Ward Peeters <[email protected]>
lmdb.remove() is asynchronous so getNode can still return deleted nodes for a time until the delete operation completes.
Switching to removeSync fixes this (at the cost of some amount of perf penalty).