Support custom DNS resolution

[LucioFranco]

As we start to support more cloud native environments users will probably want to use custom DNS resolution to find downstream services. An example of this would be using consul's DNS server to find some downstream sink. Vector should have this support out of the box.

Use case

As a user, I want vector to be able to find downstream services using custom DNS resolution configuration.

Solution

Allow users to supply custom DNS servers or configurations and let Vector pick them up so that they can be used when attempting to connect to a downstream service. In environments like k8 they expose their services via DNS and it is a native protocol for doing service discovery. This will become increasingly useful as Vector gets deployed in more complex cloud environments.

[binarylogic]

@LucioFranco can you provide more detail on this? Which sinks would need this? It's not clear on this issue what work actually needs to be done.

[LucioFranco]

@binarylogic I updated the description should be more clear now.

[ktff]

Implementation

Core

Add custom name resolver which vector should use on top of the current one. This resolver would be configured by new global option

    # IP:port of DNS servers. If name resolution with specified servers fails,  
    # OS resolver will be used. If port is not specified, it's assumed to be 
    # DNS port 53.
    # 
    # * optional
    dns = ["1.1.1.1:53","8.8.8.8"]

Resolver will query DNS servers for A and AAAA records.

It would be good to have all name resolutions done by this resolver to provide simpler UX. This would require modifying sinks: elasticsearch sink, http sink, splunk_hec sink, clickhouse sink, vector sink, kafka sink, and tcp sink. For that, SinkConfig::build method could be changed to accept additional configuration as currently there is no clean way of providing additional configuration to Sinks. GlobalOptions is a good candidate to pass. And of sources, kafka source needs to be modified. And any other occurrence of name resolution that I've maybe missed.

Extensions

There are ways to make sink/source discovery even more user friendly in environments like Kubernetes for which this is a good base on which they could be individually added.

[ktff]

@LucioFranco @binarylogic could you review this.

[LucioFranco]

@ktff I think this makes sense! For the latter part we probably want to update the connector, I think @bruceg change recently touched that part.

[binarylogic]

@ktff this looks good to me. What would the default be for this? I assume we'd offer DNS resolution by default, without any configuration, correct?

[binarylogic]

And to clarify, let's proceed with this. Feel free to begin work unless you have any other outstanding questions or blockers.

[ktff]

What would the default be for this? I assume we'd offer DNS resolution by default, without any configuration, correct?

@binarylogic Yes, default DNS resolution will be that of OS, while the provided DNS servers will have priority over OS.

[LucioFranco]

@lukesteensen do you think we should bundle some of this with more consolidation of our http client. Like removing the default rusoto client and instead use a custom hyper client with our own connector so that we can provide our own DNS resolution? Not sure how worth it might be.

[lukesteensen]

I don't have a good sense of how much work that would be. I'm also not sure how often people will want custom DNS resolution of AWS services.

[ktff]

@LucioFranco I don't think there is a need for that, since not all DNS resolutions are done over http client. And changing HttpService to use different resolver is a small task.

[LucioFranco]

@ktff you're right, we should though implement the DNS resolver as a tower service. I know hyper will be adding support for this in 0.13.

@lukesteensen its pretty much this https://github.com/LucioFranco/tower-rusoto/blob/master/src/lib.rs but it would be nice for us to all use one http client throughout all of vector imo.

[ktff]

we should though implement the DNS resolver as a tower service

@LucioFranco do you mean implementing hyper::client::connect::dns::Resolve trait?

[LucioFranco]

Yes, but that is moving to become a tower_service in hyper 0.13 hyperium/hyper#1903

[ktff]

I think it isn't worth the wait. Since they will be just aliasing it, updating implementation to this new version will be easy. I can also write this so that only HttpService depends on trait Resolve. Then when dependency to hyper is updated to 0.13, this could be updated as well.

[LucioFranco]

Oh not suggesting we move, but thinking we may just want to towerize it anyways.

[ktff]

I agree, then I'll implement this as a Service with Resolve shim on top of it to interface with the http client.

[LucioFranco]

Update: Once #1118 is merged, we can start work on reworking our internal traits to support passing down the resolver. This will allow our sinks to use it as either a HttpConnector or use it to manually resolve ips.

[binarylogic]

Closing this since #1118 resolved the core issue. We will be opening a follow up issue for each sink's implementation.