Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[新需求]针对post请求，相同的url，根据参数不同返回不同结果的页面抓取实现 #15

Open
zhaoxin1124 opened this issue Dec 25, 2018 · 1 comment
Open

[新需求]针对post请求，相同的url，根据参数不同返回不同结果的页面抓取实现 #15

zhaoxin1124 opened this issue Dec 25, 2018 · 1 comment

Comments

@zhaoxin1124
Copy link

针对post请求，相同的url，根据参数不同返回不同结果的页面抓取实现
是否可考虑在解析页面结果的类中返回当前爬虫对象，这样可以在处理完上一个页面抓取后，向爬虫对象中的url队列添加新的url。增强现在的只能在爬虫初始化的时候添加url（或者只能粗犷的扩散爬取）功能。
@1988tianyuan
Copy link

可以让PageParser获取RunData引用，在parse的时候动态添加url：
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zhaoxin1124 @1988tianyuan