Store request info in WARC

[jnioche]

we currently deal with the response only

[jnioche]

@sebastian-nagel assigning to you but I can do it when I am back from hols

[jnioche]

Couldn't find a way of accessing the complete request from httpclient so we'll have to do with the response only.

[jnioche]

Instead we can store the http header stuff in the warcinfo section

    String userAgent = AbstractHttpProtocol.getAgentString(getConf());
        fields.put("http-header-user-agent", userAgent);
        fields.put("http-header-from",
                ConfUtils.getString(getConf(), "http.agent.email"));

        byte[] warcinfo = WARCRecordFormat.generateWARCInfo(fields);

see [https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/#warcinfo]

[sebastian-nagel]

Indeed, information in request records is mostly redundant:

• host, IP address, requested URI and date are also contained in the response record
• user agent info to be added to warcinfo
• also accept params (Accept, Accept-Language, Accept-Encoding) are constant for all requests
  Need to check whether WARC tools and libraries do not choke without request records.

[sebastian-nagel]

Closing. This has been implemented in stormcrawler#691.