feat: detect website fragments

[kemingy]

• close bug: anchor/fragment detection doesn't appear to work #1457

[kemingy]

• Cargo clippy errors are fixed in another PR: Specify MSRV #1676

[mre]

Nice pull request! Only had one improvement idea.

[mre]

Looks good to me!
We should add a test or two to make sure it works. I guess the easiest would be an integration test in lychee-bin/tests/cli.rs. But I wouldn't mind merging this already, in case you won't find the time to add it. Just let me know. 😉

[kemingy]

Looks good to me! We should add a test or two to make sure it works. I guess the easiest would be an integration test in lychee-bin/tests/cli.rs. But I wouldn't mind merging this already, in case you won't find the time to add it. Just let me know. 😉

Added to the test cases for both markdown and HTML.

TIL, GitHub HTML has a "user-content-" prefix for fragments added in the README file.

[mre]

Yes, great! One final rebase, and we should be good to go. 😄

[kemingy]

Yes, great! One final rebase, and we should be good to go. 😄

Rebased. PTAL

[mre]

Fantastic! Thank you.

[honzajavorek]

This has introduced several broken links on my website. It's because they point to websites where the fragment is handled by JavaScript, so the link does work, but Lychee can't see it. It would be awesome if I could add something like "ignore fragments checking for these few links where I manually verified it's actually okay" option to my TOML 🙏 Without it I'll have to turn off checking fragments altogether, and that would be a shame.

[honzajavorek]

As an example, https://github.com/lycheeverse/lychee#readme works in the browser, but Lychee will deem it as broken fragment. That's because GitHub uses React nonsense to render the page.

[kemingy]

As an example, https://github.com/lycheeverse/lychee#readme works in the browser, but Lychee will deem it as broken fragment. That's because GitHub uses React nonsense to render the page.

I think we can add some special logic to handle the GitHub Markdown fragments. (lychee already has some special logic for GitHub in the code)

We could detect if the URL contains "github.com" then try the "user-content-" prefix for the fragments.

I have checked that GitHub gist also follows this rules.

What do you think? @mre

[honzajavorek]

That’s awesome for GitHub, but I have such links from other sites, too.

…

On Thu 15. 5. 2025 at 12:32, Keming ***@***.***> wrote: *kemingy* left a comment (lycheeverse/lychee#1675) <#1675 (comment)> As an example, https://github.com/lycheeverse/lychee#readme works in the browser, but Lychee will deem it as broken fragment. That's because GitHub uses React nonsense to render the page. I think we can add some special logic to handle the GitHub Markdown fragments. (lychee already has some special logic for GitHub in the code) We could detect if the URL contains "github.com" then try the "user-content-" prefix for the fragments. I have checked that GitHub gist also follows this rules. What do you think? @mre <https://github.com/mre> — Reply to this email directly, view it on GitHub <#1675 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACFGMNCRT7BMA3TT5JLAWT26RUNTAVCNFSM6AAAAAB256NQFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQOBTGM2DGNZTGE> . You are receiving this because you commented.Message ID: ***@***.***>

[mre]

It would be awesome if I could add something like "ignore fragments checking for these few links where I manually verified it's actually okay" option to my TOML

I guess you can?

❯❯❯ echo 'https://github.com/lycheeverse/lychee#readme' | lychee -vvv --exclude 'https://github.com/lycheeverse/lychee#' -
[EXCLUDED] https://github.com/lycheeverse/lychee#readme

🔍 1 Total (in 0s) ✅ 0 OK 🚫 0 Errors 👻 1 Excluded

The trick is to add a # at the end of the site you want to exclude.

Also, at least for me, your example link does work:

❯❯❯ echo 'https://github.com/lycheeverse/lychee#readme' | lychee -vvv -
     [200] https://github.com/lycheeverse/lychee#readme

🔍 1 Total (in 0s) ✅ 1 OK 🚫 0 Errors

Can somebody double-check?

If it turns out to be an issue, it would be better to create a new issue referencing this pull request instead.

[honzajavorek]

I can exclude the links completely, but then they're not checked if they're 404.

Also, at least for me, your example link does work

That's interesting. I didn't have time to dig into this for the past week. I'll get back to it soon and if there's a feature to request, I'll request it as a separate issue. Thanks for Lychee and thanks for replying here 🙏