Search results for "UTF-8 with BOM" files shifted on first line by a character

[Deleted user (ghost)]

Fix #66188

This is my first pull request in this project. Please let me know if my solution is not good enough, I am willing to improve it.

[Microsoft Contribution License Agreements (msftclas)]

All CLA requirements met.

[roblourens]

Why this?

[Deleted user (ghost)]

If it's a UTF-8 with BOM file and _fullPreviewRange.startLineNumber is 0, _fullPreviewLines[0] will starts with BOM, then _fullPreviewLines does not match _fullPreviewRange.

[roblourens]

But the BOM should be stripped already by ripgrepTextSearchEngine right?

[Deleted user (ghost)]

I can move it to ripgrepTextSearchEngine.

[roblourens]

But it's already there, right? That file should be stripping the BOM correctly in all cases with your change.

[roblourens]

Why change this? The BOM is 3 bytes long.

[Deleted user (ghost)]

https://github.com/Microsoft/vscode/blob/7c8361ef698d9ed491612cd786952aba2ab47c87/src/vs/workbench/services/search/node/ripgrepTextSearchEngine.ts#L258

Because there is toString() here, Buffer.from([0xEF, 0xBB, 0xBF]).toString().length is 1.

[roblourens]

Ok, that's correct thanks.

[Deleted user (ghost)]

I found another solution: Remove && options.encoding !== 'utf8'

https://github.com/Microsoft/vscode/blob/f0f3b922bcb081c6488b7299dbef4076a1cfde82/src/vs/workbench/services/search/node/ripgrepTextSearchEngine.ts#L333

then ripgrep will not output BOM character in JSON.

Or I can manually strip the BOM character in ripgrepTextSearchEngine.

Which do you prefer?

[roblourens]

I want to keep that because I found that search was faster in some cases without it. Also, you could probably search with a different encoding but still end up finding results in UTF8 files with BOMs. Also I don't trust ripgrep to keep that behavior forever. So, let's keep the check to strip the BOM on our end.

[Deleted user (ghost)]

I stripped fullText at
https://github.com/Microsoft/vscode/blob/c22caf616a9d6baa90ab904292bddc14a7b09ffd/src/vs/workbench/services/search/node/ripgrepTextSearchEngine.ts#L285
So it's not need to edit searchModel.ts.

[roblourens]

Looks great, thanks!