logging: Implement Caddyfile support for filter encoder

[francislavoie]

Closes #3694

This is essentially done but lacks tests and proper support for the ip_mask filter.

Caddyfile:

:80

log {
	output stdout
	format filter {
		wrap console
		fields {
			request>headers>Authorization delete
			request>headers>Server delete
			request>remote_addr ip_mask {
				ipv4 24
				ipv6 32
			}
		}
	}
}

Adapted JSON:

{
	"logging": {
		"logs": {
			"default": {
				"exclude": [
					"http.log.access.log0"
				]
			},
			"log0": {
				"writer": {
					"output": "stdout"
				},
				"encoder": {
					"fields": {
						"request\u003eheaders\u003eAuthorization": {
							"filter": "delete"
						},
						"request\u003eheaders\u003eServer": {
							"filter": "delete"
						},
						"request\u003eremote_addr": {
							"filter": "ip_mask",
							"ipv4_cidr": 24,
							"ipv6_cidr": 32
						}
					},
					"format": "filter",
					"wrap": {
						"format": "console"
					}
				},
				"include": [
					"http.log.access.log0"
				]
			}
		}
	},
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						":80"
					],
					"logs": {
						"default_logger_name": "log0"
					}
				}
			}
		}
	}
}

[mohammed90]

Is the \u003e part of the actual adapter output or an artifact of copy-paste/console output?

request​\u003e​headers​\u003e​Authorization

[francislavoie]

Yeah I found it strange, I think it's just Golang's JSON encoder being funky. I don't do anything special here. I asked Matt and he said he's not concerned about it.

[mholt]

Apparently we could switch to an Encoder instead of json.Marshal if we don't want the HTML escaping:

String values encode as JSON strings coerced to valid UTF-8, replacing invalid bytes with the Unicode replacement rune. So that the JSON will be safe to embed inside HTML <script> tags, the string is encoded using HTMLEscape, which replaces "<", ">", "&", U+2028, and U+2029 are escaped to "\u003c","\u003e", "\u0026", "\u2028", and "\u2029". This replacement can be disabled when using an Encoder, by calling SetEscapeHTML(false).

— https://golang.org/pkg/encoding/json/#Marshal

To clarify, however, I'm indifferent about this. It's technically correct as-is. I don't think this will be injected into HTML script tags, so I'm also not worried about that. It's probably fine either way.

[mholt]

LGTM so far!

[mholt]

When you have a chance, I'm hoping to do a final pass for 2.2 reviews next week. :)

[francislavoie]

Okay, this now supports IP filters. I also added support for CIDR in the format that ParseCIDR supports (which tbh required some JSON gymnastics lol, but whatever, I think it's kinda useful, even if the IP portion is thrown away).

:80

log {
    output stdout
    format filter {
        wrap console
        fields {
            request>remote_addr ip_mask {
                ipv4 24
                ipv6 2001:db8::/32
            }
        }
    }
}

{
  "logging": {
    "logs": {
      "default": {
        "exclude": [
          "http.log.access.log0"
        ]
      },
      "log0": {
        "writer": {
          "output": "stdout"
        },
        "encoder": {
          "fields": {
            "request\u003eremote_addr": {
              "filter": "ip_mask",
              "ipv4_cidr": 24,
              "ipv6_cidr": "2001:db8::/32"
            }
          },
          "format": "filter",
          "wrap": {
            "format": "console"
          }
        },
        "include": [
          "http.log.access.log0"
        ]
      }
    }
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":80"
          ],
          "logs": {
            "default_logger_name": "log0"
          }
        }
      }
    }
  }
}

I still need to add a few adapt tests then this should be done.

A task for tomorrow.

[francislavoie]

Okay, this is all done.

I got rid of that abstraction I made that allows stuff like 2001:db8::/32 because frankly that's pretty dumb. I was working under incorrect assumptions with what the int for CIDR was at first (I thought it was itself the mask, as an int), which is why I went down the rabbithole of implementing that string thing. But it's not useful.

I kept the optimization I made of parsing the int into an IP mask at provision time, it's a super tiny optimization but it removes that overhead from every request being logged, so I think it's good to have. Small wins.

[mholt]

Kewl, LGTM. Shall we merge this?

[francislavoie]

Yes 😄

[utilitynerd]

A big thank you to @francislavoie for implementing this and @mholt for getting it merged!