Documentation for setting up a hyperbahn router

[sashahilton00]

Forgive me if I have completely missed the underlying concepts of this, as I have yet to fully understand how the Hyperbahn routing mesh works. It appears to me as though Hyperbahn runs on a number of servers which work to form a consistent hash ring in much the same way as ringpop to handle or forward requests to the server which has information on how to connect to the relevant servers which advertise the given servicename. The problem I am having is twofold; firstly, there seem to be several references to Hyperbahn routers, yet little to no documentation or examples detailing what is required to set up a Hyperbahn router that other services can advertise on (such as this in the docs: https://github.com/uber/tchannel-node/blob/master/docs/GUIDE.md#setting-up-hyperbahn). The second problem I am having, which is very much an individual problem, is that I am not entirely clear on how TChannel/Ringpop/Hyperbahn are related. From what I understand, TChannel is the transport mechanism used by Ringpop to essentially handleorproxy requests to the correct app instance, and return the response. Hyperbahn is built upon Ringpop, and extends it to essentially create a list of ip:ports that correspond to a given servicename. Based on the documentation, in order to use a service on Hyperbahn, one instantiates a server that advertises with servicename on Hyperbahn, and makes a request to other servicenames using the getClientChannel() method, such as (ie. Hyperbahn cannot be a client alone, it must also advertise a service). What I would be extremely grateful for would be for someone to point me in the direction of the documentation/example that demonstrates making a request over Hyperbahn and receiving the response, setting up a (barebones) Hyperbahn router, and a little detail on how tchannel subchannels and Hyperbahn interact given hyperbahn takes the root tchannel and advertises a service, how does one call different tchannel subchannels over Hyperbahn, or is this not possible/intended? And finally for someone to tell me if what I have said is utterly wrong and why. Whoever clarifies this for me shall have a beer bought for them in the event that we meet someday :)

[Raynos]

@sashahilton00

I shared some of our internal docs:

• Hyperbahn overview google
• Hyperbahn overview github

there seem to be several references to Hyperbahn routers, yet little to no documentation or examples detailing what is required to set up a Hyperbahn router that other services can advertise

I exported the "running locally" document:

• https://github.com/uber/hyperbahn/blob/master/docs/running-locally.md

As for how to run it in production:

• Set up a ringpop bootstrap file and configure it ( see config/production.json )
• Set up a remote config file and configure it ( see config/production.json )
• Bootstrap your applications with a file of all the hyperbahn hosts.

You can deploy the application, As you can see in the Makefile, it needs a host, port & bootstrapFile ( which is used by ringpop )

What I would be extremely grateful for would be for someone to point me in the direction of the documentation/example that demonstrates making a request over Hyperbahn and receiving the response, setting up a (barebones) Hyperbahn router

I just updated the tchannel-node guide. The missing piece was the running-locally document.

little detail on how tchannel subchannels and Hyperbahn interact given hyperbahn takes the root tchannel and advertises a service. how does one call different tchannel subchannels over Hyperbahn, or is this not possible/intended?

Hyperbahn takes the rootChannel so it can call channel.address() and advertise the host & port to hyperbahn.

For each service you want to call you call getClientChannel({ serviceName: '...' }) and use that to make outbound requests.

So each process will advertise with hyperbahn and create N client channels to make outbound requests. The root channel is used to handle inbound requests.

And finally for someone to tell me if what I have said is utterly wrong and why. Whoever clarifies this for me shall have a beer bought for them in the event that we meet someday :)

As a bonus check out:

• Hyperbahn & tchannel video
• Ringpop talk

I'm afraid there are more internal docs that are harder too share.

My personal experience with running ~300k million QPS through hyperbahn @ uber has been that setting it up isn't too difficult, we were able to deploy and get ~10k QPS on it in a few weeks with confidence but scaling it out is more difficult.

The hyperbahn system was designed for "easy service discovery" and "tight coupling with tchannel RPC". Once hyperbahn is running, services are self service and you don't have to do any configuration.

To aid with scaling out we have an internal hyperbahn forks that contains some "useful" additions like dashboards and nagios alerts that is closed source. With these it will be a lot easier to monitor and have confidence in hyperbahn. We also have internal kafka -> elk integration.

[Raynos]

@sashahilton00

I should also give a warning, the hyperbahn service discovery & routing solution is built by Uber, for Uber.

The open source project is poorly documented and running hyperbahn is non-trivial. I do not recommend using hyperbahn as an off the shelf solution for service discovery & routing.

There are better open source solutions available.

However, since the hyperbahn project is open source, if anyone wishes to implement their own router or service discovery system in nodejs or any language, then this project is a great reference implementation to study. Especially the integration tests explain a lot of scenarios in detail.

[sashahilton00]

@Raynos Thanks very much for taking the time to lay that out for me, much appreciated. Just on the points you mentioned:

Set up a ringpop bootstrap file and configure it ( see config/production.json )
Set up a remote config file and configure it ( see config/production.json )
Bootstrap your applications with a file of all the hyperbahn hosts.

I assume the ringpop bootstrap file is a a JSON array of the host:ports acting as hyperbahn routers. If this is the case, what is included in the remote-config.file seen referenced here: https://github.com/uber/hyperbahn/blob/master/config/production.json#L12

Bootstrap your applications with a file of all the hyperbahn hosts.

I assume you mean passing a seedlist of the hyperbahn routers to tchannel as mentioned in the document you linked to. Will tchannel connect to all hyperbahn routers or just connect to a random few? Can one just pass a file with an array of all the Hyperbahn router host:ports?

Hyperbahn takes the rootChannel so it can call channel.address() and advertise the host & port to hyperbahn.

I was discussing an issue related to this here: uber-node/ringpop-node#290 (comment) (from this comment to the bottom). I was essentially asking the question of how does HyperbahnClient.advertise() work when running in docker, given that docker maps an ephemeral port to the internal listening port, the ephemeral port of which is unknown to the node application.

For each service you want to call you call getClientChannel({ serviceName: '...' }) and use that to make outbound requests.

I assume that using this will just open a line of communication to one of the servers advertising on that serviceName. If so, I assume I need to run Ringpop on each of the servers advertising on a Hyperbahn serviceName to handleOrProxy to the correct server within that particular service (I've had a few beers, sorry if that is unclear).

Thanks for the links to the video, I have actually watched them (multiple times, kudos to @mranney for that presentation, refreshingly concise) to get a better understanding of how Hyperbahn/Ringpop works, as the concept of decentralised service discovery based on hashing is much more appealing when compared to having a centralised service registry. We are just running an etcd registry at the moment whilst we build the services.

Also, thanks for the warning, I was thinking that I was just being thick and that the answer lay in the documentation in plain sight. We are currently in the process of testing possible architectures for rewriting our prototype for production, and hyperbahn looks like a pretty good starting point for service discovery on an architecture that is mostly node.js at the moment, albeit it comes with a relatively steep learning curve. If you know of any better service discovery systems that are aimed at node.js, please drop a link, we have been looking at the likes of eureka, serf and so on, though if you have any in mind, I'm listening. Other than that, thanks for the help you have provided thus far.