[Self-Host]

[HisenXuEdu]

I'm getting different results when scraping the same webpage using my self-deployed service versus the playground. Both are using version 2 of the scrape function. I copied the scraping code from the playground, but my local deployment retrieves the mobile version of the webpage, while the playground retrieves the desktop version.

I'd like to know if the self-deployed code is consistent with the playground's implementation, and how I can modify my local setup to retrieve the desktop version of the content.

[amplitudesxd]

This is to be expected. The hosted version of Firecrawl uses Fire Engine, which the self hosted version doesn't provide. You can see more details on differences at https://docs.firecrawl.dev/contributing/open-source-or-cloud

[HisenXuEdu]

How can I make the self-hosted version return the desktop version of the webpage content? My settings are as follows:

{
"url": "https://foodtechx.com/",
"onlyMainContent": true,
"maxAge": 172800000,
"formats": [
"html"
],
"waitFor": 1000,
"headers": {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
}
}

The access is made to http://localhost:3002/v2/scrape

[harshithvh]

@amplitudesxd Just to clarify, is this difference between the cloud and self-hosted versions of Firecrawl a known and intentional one?
If not, is there any plan to bring this feature to the self-hosted version in the future?

[amplitudesxd]

@harshithvh Yes, this is an intentional decision. At the moment, there's no plans to bring Fire Engine to the self-hosted version.

[amplitudesxd]

@HisenXuEdu Firecrawl should default to the Desktop version of a site!

[HisenXuEdu]

@harshithvh I'm using chromedp for scraping and need to adjust the window-size to get the desktop version of webpage content. Can Firecrawl adjust window-size for this purpose?

[harshithvh]

@HisenXuEdu Firecrawl does not provide a built-in option to set or adjust the browser window size (such as width and height) during scraping
If you specifically need to control the window size to get the desktop version of a webpage, you would need to use chromedp directly, where you can set the viewport or window size as needed.

The team can consider it for future updates.
ccing: @mogery @nickscamara

[HisenXuEdu]

@harshithvh Thank you very much for your reply！