Improve wikilink parsing and checking

[JayJayArr]

• Extract the correct filename from a Wikilink by removing potholes and fragments
• Create a new Checker for wikilinks, make it clear to the users that a base-url is required
  • Check Obsidian style wikilinks without headers
  • strip fragments & potholes from wikilinks
  • traverse the wiki-directory and check if any of the Filenames exist with any of the specified --fallback-extensions

[JayJayArr]

@jrfnl I am having trouble understanding the MediaWiki links with () and :.
Are there really markdown files named Help:Links or where do they point to?
Is [[Extension:DynamicPageList (Wikimedia)]] different to [[Extension:DynamicPageList]]

Could you please clarify this?

[mre]

Guess you can rebase on top of master now to fix CI.

[mre]

Good work! I've added a bunch of comments, but don't worry.
Most of them are about very minor paper cuts. Most of it can be fixed pretty easily by clicking on the "accept suggestion" button, I hope. 😉

[mre]

@JayJayArr, any chance you could fix the merge conflicts? I would love to get this great work merged. 😊

[JayJayArr]

There we go. Lets call it okay for now 😅

[JayJayArr]

Apparently archive.org has some problems right now which breaks CI

[JayJayArr]

Thanks for taking a look at the error handling here @thomas-zahner !
Added some more testcases for valid and invalid WikiLinks, if you still have any specific testcase in mind that we should check here please let me know. Added a check for unsupported characters as well 😅

[thomas-zahner]

Thank you for the additions. We're almost ready to merge. Apart from the review comments I still don't see any tests which check for invalid wikilinks. Maybe I've missed it?

Could you add a test which results in a WikilinkNotFound error. This path seems to be untested as of now.

[JayJayArr]

Added a Test from the cli and a Unit Test for the WikilinkResolver for this case. You did not miss it, it was indeed missing, sorry for the gap there!

[thomas-zahner]

@JayJayArr Thank you very much for the changes and the work, this really is a great addition. I've just removed a dbg! expression you probably forgot to remove and simplified indexing with 0c3dcb5.

[jrfnl]

@JayJayArr and everyone who reviewed and contributed to the discussion: thank you for working on this! I look forward to seeing this in the next release. 💞

[JayJayArr]

Thanks to both of you @mre & @thomas-zahner for your support on this!

[katrinafyi]

I was trying out this feature and I wrote these notes about things that I thought were unexpected:

• when a link is not found, the error message is long because it repeats the message twice. it also repeats the base directory twice and makes the invalid path /home/x/progs/lychee/fixtures/home/x/progs/lychee/fixtures/Doesn't exist.md - i don't know whether this is intentional.

     [ERROR] file:///home/x/progs/lychee/fixtures/Doesn't%20exist.md |
     Wikilink file:///home/x/progs/lychee/fixtures/Doesn't%20exist.md not found at /home/x/progs/lychee/fixtures/home/x/progs/lychee/fixtures/Doesn't exist.md:
     WikiLink file:///home/x/progs/lychee/fixtures/Doesn't%20exist.md could not be found at /home/x/progs/lychee/fixtures/home/x/progs/lychee/fixtures/Doesn't exist.md
  

• wikilinks always ignore fragments? i can't see anything in the obsidian help page which says they should be disregarded. the unit tests also seem to say that they should be resolved (unless "resolve" is meant as "ignore"). i note that if fragments were to be processed, the current markdown extractor would have to be changed. atm, it makes anchors using #kebab-case which is different to what obsidian uses in wikilinks.

• wikilinks, if enabled, are tried on all file:// links, including absolute links. for instance, a link like [a](file:///tmp/Usage.md) could succeed when /tmp/Usage.md doesn't exist and a Usage.md exists inside the base folder. maybe it should be limited to relative links which only have filenames.

• the wikilink feature does its own directory walk. this is different from other directory walks in lychee and won't respect options like gitignore or hidden files. actively ignoring symlinks is also weird.

• to specify the indexing folder, i think --root-dir would've made more sense, or it could even be based off the inputs provided through command line. but it does work with --base-url so it's nbd.

Aside from the first point, I think these points are kinda hard to fix or don't necessarily need fixing. Just things to think about.