Request help using page.SetExtraHeaders to set 'Host' header to do vhosted screenshots.

[CyberStryk]

Hi! I've been working on a fork of projectdiscovery's httpx to try and get screenshots working for vhosted hosts.

I'm trying to accomplish this using the page.SetExtraHeaders function using the line below:

	_, vhosterr := page.SetExtraHeaders([]string{"Host", "test.com"})
	if vhosterr != nil {
		return nil, "", vhosterr
	}

None of the requests my server are getting 'Host: test.com' header set properly. Any ideas what I'm doing wrong? Is there an easier way to do this? I've gotten vhosting to work with launcher.New() options, but I'd rather not create a new browser for each request.

Here is the screenshot function:

func (b *Browser) ScreenshotWithBody(url string, timeout time.Duration) ([]byte, string, error) {

	page, err := b.engine.Page(proto.TargetCreateTarget{})
	if err != nil {
		return nil, "", err
	}
	// use page host header to do virtual host here:
	_, vhosterr := page.SetExtraHeaders([]string{"host", "test.com"})
	if vhosterr != nil {
		return nil, "", vhosterr
	}

	page = page.Timeout(timeout)
	defer page.Close()

	if err := page.Navigate(url); err != nil {
		return nil, "", err
	}

	page.Timeout(5 * time.Second).WaitNavigation(proto.PageLifecycleEventNameFirstMeaningfulPaint)()

	if err := page.WaitLoad(); err != nil {
		return nil, "", err
	}
	_ = page.WaitIdle(1 * time.Second)

	screenshot, err := page.Screenshot(true, &proto.PageCaptureScreenshot{})
	if err != nil {
		return nil, "", err
	}

	body, err := page.HTML()
	if err != nil {
		return screenshot, "", err
	}

	return screenshot, body, nil
}

[ysmood]

Have you tried this?

rod/hijack_test.go

Line 46 in 938f47b

r.Req().Header.Set("Test", "header") // override request header

[CyberStryk]

Thanks for the response!

I've been trying to implement the Hijack method, but I'm very much struggling to get it to work. Main thing I'm struggling with is how to configure the router to Hijack my page's request. I've tried following along from the referenced hijack_test.go and I'm getting too lost in the weeds understand how exactly I get my browser/page to route through it.

If you're able to provide any more help I'd greatly appreciate it!

func (b *Browser) ScreenshotWithBody(url string, timeout time.Duration) ([]byte, string, error) {

	page, err := b.engine.Page(proto.TargetCreateTarget{})
	if err != nil {
		return nil, "", err
	}

	page = page.Timeout(timeout)
	defer page.Close()

	// Setting extra headers with a HijackRequest.
	router := page.HijackRequests()
	defer router.MustStop()

	router.MustAdd("*", func(ctx *rod.Hijack) {
		r := ctx.Request.SetContext(page.GetContext())
		r.Req().Header.Set("Host", "hijacked.host.header")
	})
	// End setting extra headers

	if err := page.Navigate(url); err != nil {
		return nil, "", err
	}

       // ... code continues.

[ysmood]

https://go-rod.github.io/#/network/README?id=hijack-requests

[CyberStryk]

Yeah, I've tested a bunch of iterations of what's in the README and the tests. I've tried hijacking the browser, page, and both, but I can't get it to edit the request that gets sent to the server. I'm probably missing something simple.

Page.SetExtraHeaders and Page.MustSetExtraHeaders both work adding new headers but not for setting "Host" header. The only thing that has worked for me is setting a resolver in the launcher options which I'm trying to avoid.

package main

import (
	"github.com/go-rod/rod"
	"github.com/go-rod/rod/lib/proto"
)

func main() {
	page := rod.New().MustConnect().MustPage("")
	router := page.HijackRequests()

	router.MustAdd("*", func(ctx *rod.Hijack) {
		ctx.Request.Req().Header.Set("host", "hijack.request")
		ctx.ContinueRequest(&proto.FetchContinueRequest{})

	})

	go router.Run()

	page.MustNavigate("http://127.0.0.1/").MustWaitStable().MustScreenshot("")

}

[ysmood]

cdp protocol doesn't allow you to change host with SetExtraHeaders. It's a upstream issue, which you might have to contact chromium team.

Hijack or proxy works fine to me.

[CyberStryk]

I finally got it to work! ctx.Request.Req().Header.Set() function does not work for setting the "host" header. It only works for all the other headers.

To set the "host" header you must use the ctx.Request.Req().Host variable.

Thank you @ysmood for taking the time to help me.

package main

import (
	"github.com/go-rod/rod"
)

func main() {
	page := rod.New().MustConnect().MustPage("")
	router := page.HijackRequests()

	router.MustAdd("*", func(ctx *rod.Hijack) {

		// To set the host Header use Req().Host:
		ctx.Request.Req().Host = "test.host"

		// DOES NOT WORK for setting host
		ctx.Request.Req().Header.Set("Host", "does.not.set.host")

		// To set any other headers use the Req.Header.Set() function
		ctx.Request.Req().Header.Set("User-Agent", "test-agent")
		ctx.Request.Req().Header.Set("Accept-Language", "test-language")
		ctx.Request.Req().Header.Set("Accept", "test/html")
		ctx.Request.Req().Header.Set("Upgrade-Insecure-Requests", "test-insecure")
		ctx.Request.Req().Header.Set("Accept-Encoding", "test-encoding")

		// Load the response using the modified request
		ctx.MustLoadResponse()

	})

	go router.Run()

	page.MustNavigate("http://127.0.0.1/")

}

└─$ sudo nc -nvlp 80
Listening on 0.0.0.0 80
Connection received on 127.0.0.1 35342
GET / HTTP/1.1
Host: test.host
User-Agent: test-agent
Accept: test/html
Accept-Encoding: test-encoding
Accept-Language: test-language
Upgrade-Insecure-Requests: test-insecure

[CyberStryk]

Page.SetExtraHeaders does not allow for modifying the host header.

To set the "Host" request header use the Request.Req().Host variable.

To set any other request headers use the Request.Req().Header.Set() function.

Example:

package main

import (
	"github.com/go-rod/rod"
)

func main() {
	page := rod.New().MustConnect().MustPage("")
	router := page.HijackRequests()

	router.MustAdd("*", func(ctx *rod.Hijack) {

		// To set the host Header use Req().Host:
		ctx.Request.Req().Host = "test.host"
		// DOES NOT WORK for setting host
		ctx.Request.Req().Header.Set("Host", "does.not.set.host")

		// To set any other headers use the Req.Header.Set() function
		ctx.Request.Req().Header.Set("User-Agent", "test-agent")
		ctx.Request.Req().Header.Set("Accept-Language", "test-language")
		ctx.Request.Req().Header.Set("Accept", "test/html")
		ctx.Request.Req().Header.Set("Upgrade-Insecure-Requests", "test-insecure")
		ctx.Request.Req().Header.Set("Accept-Encoding", "test-encoding")

		// Load the response using the modified request
		ctx.MustLoadResponse()

	})

	go router.Run()

	page.MustNavigate("http://127.0.0.1/")

}

Example output:

┌──(username㉿hostname)-[~]
└─$ sudo nc -nvlp 80
Listening on 0.0.0.0 80
Connection received on 127.0.0.1 35342
GET / HTTP/1.1
Host: test.host
User-Agent: test-agent
Accept: test/html
Accept-Encoding: test-encoding
Accept-Language: test-language
Upgrade-Insecure-Requests: test-insecure