shard splitting doesn't always respect nested docs

[s1monw]

Nested docs share the same _id which is enough unless _routing is involved and / or the index is partitioned. We need to find a good way to respect nested docs when we select the docs that should be deleted. We can either do this at runtime (when we split) or we also index / store the _routing value from the parent.

@jpountz WDYT

[martijnvg]

I think if the mapping has nested fields then we should just wrap the ShardSplittingQuery in a ToChildBlockJoinQuery? This should include the document that do not belong in the current shard with their nested documents.

[s1monw]

@mvg this sounds like a good plan, I wonder if we then need to skip all child docs when we select documents for deletion? the query we pass to the ToChildBlockJoinQuery should only match parents, right?

[martijnvg]

I wonder if we then need to skip all child docs when we select documents for deletion

Right, because then there is one way of selecting nested documents.

the query we pass to the ToChildBlockJoinQuery should only match parents, right?

Yes. Then the ShardSplittingQuery query should then in the default case (no routing AND no partitioning) not select nested documents by itself.

Maybe we can create a bool query with two should clause:

• ShardSplittingQuery which selects all regular Lucene documents that do not belong on the current shard.
• ToChildBlockJoinQuery with ShardSplittingQuery as parentQuery and a bitset based on Queries.newNonNestedFilter() query as parentsFilter. This will select all nested documents of regular documents that do not belong on the current shard.

[martijnvg]

The ShardSplittingQuery can be an expensive query, so alternatively we can introduce a new query. This query would wrap the ShardSplittingQuery query and emit its docids and use BitSet (based Queries.newNonNestedFilter()) to also emit all nested docids belong to each normal document. This way only a single ShardSplittingQuery instance needs to be used.