Request headers are not honored on the redirections followed from the original resource. #132

clopez · 2020-11-03T02:32:57Z

Consider the following test program

#!/usr/bin/node
const request = require("sync-request");

function getDataFromUrl(src) {
    const res = request("GET", src, {
      headers: {
        "User-Agent": "sync-request/6.1.0"
      }});
    return res.getBody("utf8");
};

result = getDataFromUrl(process.argv.slice(2)[0]);
console.log(result);

It simply does a request to the url passed via command line argument setting the header User-Agent to sync-request/6.1.0 and prints the result.

If i run it to check the header it sends to the URL https://headers.cloxy.net/request.php everything looks ok. But when I try the URL http://headers.cloxy.net/request.php (without https) it doesn't work as expected. This is because the url with http does a re-direct to the url with https and then sync-request doesn't set the user-agent header on the final url when following the redirection from the original url.

Example:

$ node test.js https://headers.cloxy.net/request.php
<!DOCTYPE html>
<html>
	<head>
		<meta charset="utf-8" />
		<title>Show My HTTP Request Headers</title>
	</head>
	<body>
		<h1>Your Request HTTP Headers</h1>
		<ul>
			<li>Connection: close</li>
			<li>Host: headers.cloxy.net</li>
			<li>Accept-Encoding: gzip,deflate</li>
			<li>User-Agent: sync-request/6.1.0</li>
		</ul>
		<hr />
		<p>Powered by:</p>
		<p><a href="https://peername.com/" target="_blank"><img src="/peername.png" alt="PeerName Blockchain-based domains" title="Blockchain-based domains" /></a></p>
	</body>
</html>

$ node test.js http://headers.cloxy.net/request.php
<!DOCTYPE html>
<html>
	<head>
		<meta charset="utf-8" />
		<title>Show My HTTP Request Headers</title>
	</head>
	<body>
		<h1>Your Request HTTP Headers</h1>
		<ul>
			<li>Connection: close</li>
			<li>Host: headers.cloxy.net</li>
		</ul>
		<hr />
		<p>Powered by:</p>
		<p><a href="https://peername.com/" target="_blank"><img src="/peername.png" alt="PeerName Blockchain-based domains" title="Blockchain-based domains" /></a></p>
	</body>
</html>

Note how in this second request, following the redirect, it not only dropped the User-Agent header but also The Accept-Encoding one

The url at the http site does a standard 301 redirect to the site with https as shown below

$ curl -i http://headers.cloxy.net/request.php
HTTP/1.1 301 Moved Permanently
Date: Tue, 03 Nov 2020 02:28:49 GMT
Server: Apache
Location: https://headers.cloxy.net/request.php
Cache-Control: max-age=3600
Expires: Tue, 03 Nov 2020 03:28:49 GMT
Content-Length: 245
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://headers.cloxy.net/request.php">here</a>.</p>
</body></html>

The text was updated successfully, but these errors were encountered:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Request headers are not honored on the redirections followed from the original resource. #132

Request headers are not honored on the redirections followed from the original resource. #132

clopez commented Nov 3, 2020 •

edited

Loading

Request headers are not honored on the redirections followed from the original resource. #132

Request headers are not honored on the redirections followed from the original resource. #132

Comments

clopez commented Nov 3, 2020 • edited Loading

clopez commented Nov 3, 2020 •

edited

Loading