Skip to content

[Features]: Enable sitemap.xml-based crawling and following domain redirects #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[Features]: Enable sitemap.xml-based crawling and following domain redirects #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d13b01a
feat: add support for following domain redirects (disabled by default)
K-Mistele Jan 16, 2025
cc3470c
feat: add sitemap parser
K-Mistele Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ node_modules
foo.txt
*.log
.DS_Store
dist/
dist/
.idea/
19 changes: 19 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,25 @@ We use [mozilla/readability](https://github.com/mozilla/readability) to extract
sitefetch https://vite.dev --content-selector ".content"
```

### Follow domain redirects

You can opt-in to following domain redirects (e.g. from mywebsite.com to www.mywebsite.com):
```sitefetch
sitefetch https://vite.dev --follow-domain-redirects
```

### Sitemap crawler

Many websites include a special file `sitemap.xml` that lists all the pages on the website to help
search engines understand and crawl the site more efficiently. This is often done automatically for blogs,
documentation websites, and other types of sites managed through a CMS. It is usually not present for SPAs.

The sitemap crawler can be enabled through `--enable-sitemap`, although if a `sitemap.xml` is not present the site will be
crawled as normal:
```
sitefetch https://nextjs.org --enable-sitemap
```
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can it simply be:

sitefetch https://example.com/sitemap.xml

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires that you know ahead of time if the site you're trying to crawl has a sitemap which requires making an additional request, see above comment. Happy to change it if you feel strongly about it.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can just check if the input url ends with /sitemap.xml


## Plug

If you like this, please check out my LLM chat app: https://chatwise.app
Expand Down
Loading