[Bug] VSCode indexing problem

[KagaJiankui]

There is an annoying indexing problem on Windows machine running VSCode extension and Marksman binary built from latest main branch. The language server is only able to index a file after the file is manually opened with the VSCode editor, or the language server is not able to recognize the file elsewise. For all the unrecognized files, the language server will not prompt them for completion.

The problem may be mainly responsible to the processing on non-ASCII unicode characters especially CJK. Unfortunately, I am not able to reproduce the bug on the following simple test workspace, so I will provide the workspace(buggy.zip) containing my personal documents that leading to this bug.

Efforts to Reproduce

The architecture of the test workspace is:

root_dir
├── note1
│   ├── ch1
│   │   └── 1_foo_中文.md
│   └── ch2
│       └── 2_bar_中文.md
├── root.code-workspace
└── .marksman.toml

For example, consider two files under two subdirectory listed above.

<note1/ch1/1_foo_中文.md>
# Foo

foofoofoo

<note1/ch2/2_bar_中文.md>
# Bar

[[note1/ch1/1_foo_中文#Foo]]barbarbar <- This wikilink is unrecognized before manually opening 1_foo.md

Using the quick-fix to create the unrecognized file leads to an duplicate under the root directory of the workspace:

root_dir
├── note1
│   ├── ch1
│   │   └── 1_foo_中文.md
│   └── ch2
│       └── 2_bar_中文.md
├── root.code-workspace
├── 1_foo_中文.md <- Duplicate file created by the quickfix
└── .marksman.toml

System Info

VSCode version:

版本: 1.92.1 (system setup)
提交: eaa41d57266683296de7d118f574d0c2652e1fc4
日期: 2024-08-07T20:16:39.455Z
Electron: 30.1.2
ElectronBuildId: 9870757
Chromium: 124.0.6367.243
Node.js: 20.14.0
V8: 12.4.254.20-electron.0
OS: Windows_NT x64 10.0.22631

Extension versiono: 0.3.4

Server version: 1.0.0-8c07ec6+8c07ec69b927a31234316db99320adeb630a2671