[BUG] page.on('request', r => r.headers) doesn't return all headers on Chromium and Webkit

[verhovsky]

Context:

• Playwright Version: 1.9.1
• Operating System: Linux
• Node.js version: 14.14.0
• Browser: Chromium and WebKit. Possibly Firefox too

Compare the output of following:

import playwright from 'playwright';

(async () => {
  for (const browserType of ['chromium', 'firefox', 'webkit']) {
    const browser = await playwright[browserType].launch();
    const page = await browser.newPage();

    page.on('request', req => {
        console.log(req.headers());
    });
    await page.goto("https://en.wikipedia.org/wiki/Main_Page/");

    await browser.close();
  }
})();

On Chromium, you'll only get like

{
  'sec-ch-ua': '',
  referer: 'https://en.wikipedia.org/wiki/Main_Page/',
  'sec-ch-ua-mobile': '?0',
  'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4421.0 Safari/537.36'
}

On Firefox, you'll see

{
  host: 'en.wikipedia.org',
  'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0',
  accept: 'image/avif,image/webp,*/*',
  'accept-language': 'en-US,en;q=0.5',
  'accept-encoding': 'gzip, deflate, br',
  referer: 'https://en.wikipedia.org/wiki/Main_Page/',
  connection: 'keep-alive',
  cookie: 'WMF-Last-Access=<REDACTED>:v4'
}

For my use-case, I used Firefox through playwright to load a website and get a fresh cookie that I then used for scraping that website using requests. I couldn't get the cookie with Chromium. I didn't check if Firefox returns all the headers, it returns the one I cared about.

This is the puppeteer issue: puppeteer/puppeteer#4918

[pavelfeldman]

Complete set of request headers is only available upon response event. That is a browser limitation.

[verhovsky]

Oh, I see. You can change page.on('request', req => req.headers()) to page.on('response', res => res.request().headers()) and then the cookie is there.

import playwright from 'playwright';

(async () => {
  for (const browserType of ['chromium', 'firefox', 'webkit']) {
    const browser = await playwright[browserType].launch();
    const page = await browser.newPage();

    page.on('response', resp => {
        console.log(resp.request().headers());
    });
    await page.goto('https://en.wikipedia.org/wiki/Main_Page/');

    await browser.close();
  }
})();