Possibility to use playwright with scrapy

[slamer59]

Hello, some website needs js rendering.

One solution would to use as a option playwright.

Have you considered this option ?
A good option is https://github.com/scrapy-plugins/scrapy-playwright

[eliasdabbas]

Thanks @slamer59

Yes, this is a very important feature/integration to consider.
There are many use cases, and it's not a single implementation.

Do you have specific use case in mind so we can consider the details/plan?

Thanks again.

[slamer59]

Hello,
I am not sure what kind of use case you ask for but any website generated with JS rendering requires headless brother.

Reading the code and scrapy-playwright documentation, it would require to:

1. Add DOWNLOAD_HEADERS to environment (manullay or with os.env
   DOWNLOAD_HANDLERS = {
   "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
   "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
   }

2. add an attribute self.meta which by default will be an emtpy dict.
   and activated when asked.

    yield scrapy.Request("https://httpbin.org/get", meta={"playwright": True})
   

advertools/advertools/spider.py

Line 672 in 21722a5

yield Request(url, callback=self.parse, errback=self.errback)

What do you think ?

[eliasdabbas]

I meant that there are different things you might want to accomplish with JS sites, and checking for implementation.

Just checked the docs again, and it seems the main thing that needs to be added is the ability to edit the meta dictionary. This enables users to add any playwright configuration through meta and/or custom settings (this is already enabled).

So enabling meta in the adv.crawl function and passing its values to Request + custom settings should allow usage of playwright.

Just checking because I'm not very familiar with the details of playwright. Does this make sense from an implementation point of view?

If so, it can be enabled, I can release a minor version for testing, and we can check.

Let me know if you have other thoughts.

CC: @ska-ibees

[slamer59]

What you already do like:

• Extract headers / ...

DOWNLOAD_HEADERS should be set in some settings file (as I read the scrapy doc). I need further investigation.

Another implementation, if it gets more complex, is to make inheritence of this class.

[eliasdabbas]

Thanks.
I'll try a few of these things and please let me know if you have any code to test. Would love to see.