Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test for malformed disallow directives #3

Merged
merged 1 commit into from
Oct 21, 2019
Merged

Test for malformed disallow directives #3

merged 1 commit into from
Oct 21, 2019

Conversation

maramsumanth
Copy link
Contributor

@maramsumanth maramsumanth commented Oct 4, 2019 •
edited
Loading

There are other variations of useragent, sitemap, crawl delay and request rate (as mentioned in protego.py) which can be tested.

Gallaecio
Gallaecio approved these changes Oct 15, 2019
View reviewed changes
Copy link
Member

@Gallaecio Gallaecio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks!

whalebot-helmsman
whalebot-helmsman approved these changes Oct 21, 2019
View reviewed changes
Copy link
Collaborator

@whalebot-helmsman whalebot-helmsman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anubhavp28, looks good for me

@whalebot-helmsman whalebot-helmsman merged commit af3a909 into scrapy:dev Oct 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Gallaecio Gallaecio Gallaecio approved these changes

@whalebot-helmsman whalebot-helmsman whalebot-helmsman approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@maramsumanth @Gallaecio @whalebot-helmsman