Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/master' into add_marathon_watcher
Browse files Browse the repository at this point in the history 
* upstream/master:
  Fixup mistake in watcher README
  Update the README concerning ELB's weaknesses
  Fix minor Ruby 1.9 vs Ruby 2.X compat in install instructions
  Clarify that the first code block is not installing Ruby
  Update README.md with better installation directions
  Allow arbitrary haproxy_server_options to be supplied to HAProxy.
  adds specs for ec2tag watch method, and fixes bug where if an exception was raised the normal sleep got skipped, so it could get stuck calling the failing discover instances code in a tight loop every few ms
  Don't have synapse require the aws variables - credentials should be able to be nil to allow using iam instance profile. Updates tests to reflect
  allow `server_port_override` to be an int since the other `port` config options can be ints
  Linking from main README to service watcher README
  Allow pluggable watchers
  Handle path going away properly
  Do not unregister the callback in the zk watcher
  Some minor documentation fixups
  Explicit watcher haproxy backend names
  [travis] switch to new infrastructure
  bump version to v0.12.1
  bump version to v0.12.0
  Fixed wording of some documentation, added missing options
  Address feedback from igor and schleyfox
  Fixups for merge to airbnb/master
  Turns out it's important to handle session disconnects correctly
  Try out :per_callback threads and get more debug information
  Add support for the weight key added in nerve
  Fix bug in caching logic.
  Add state file.
  Rate limit restarts but not stats socket updates
  ZooKeeper connection pooling.
  Increase HAProxy restart interval.
  Revert "Add rate limiter."
  Add rate limiter.
  Allow the option allredisp option to haproxy.
  Explicitly deduplicate registrations
  Allow registrations to be manifested on the file system
  Add 'use_previous_backends' option.
  test synapse on 2.0.0, 2.1.6, 2.2.2 also
  git ignore .ruby-version
  update zookeeper dependency to support newer rubies
  typo
  synapse expects arrays under shared_frontend, hosts and backend
  verify doubles
  run transpec to convert specs to new format
  fix some broken specs and deprecation warnings
  bump rspec version

Conflicts:
	Gemfile.lock
	lib/synapse/service_watcher.rb
  • Loading branch information
@tdooner
tdooner committed Oct 10, 2015
2 parents acde0d3 + 561b384 commit aa4cbc5
Show file tree
Hide file tree
Showing 24 changed files with 764 additions and 359 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ tmp
vendor/

synapse.jar
.ruby-version
5 changes: 4 additions & 1 deletion .travis.yml
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
language: ruby
cache: bundler
sudo: false
rvm:
- 1.9.3

- 2.0.0
- 2.1.6
- 2.2.2
56 changes: 29 additions & 27 deletions Gemfile.lock
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
PATH
remote: .
specs:
synapse (0.11.1)
synapse (0.12.1)
aws-sdk (~> 1.39)
docker-api (~> 1.7.2)
zk (~> 1.9.4)
Expand All @@ -11,31 +11,31 @@ GEM
specs:
addressable (2.3.6)
archive-tar-minitar (0.5.2)
aws-sdk (1.47.0)
aws-sdk (1.64.0)
aws-sdk-v1 (= 1.64.0)
aws-sdk-v1 (1.64.0)
json (~> 1.4)
nokogiri (>= 1.4.4)
coderay (1.0.9)
crack (0.4.2)
safe_yaml (~> 1.0.0)
diff-lcs (1.2.4)
diff-lcs (1.2.5)
docker-api (1.7.6)
archive-tar-minitar
excon (>= 0.28)
json
excon (0.38.0)
excon (0.45.4)
ffi (1.9.3-java)
json (1.8.1)
json (1.8.1-java)
json (1.8.3)
little-plugger (1.1.3)
logging (1.8.2)
little-plugger (>= 1.1.3)
multi_json (>= 1.8.4)
method_source (0.8.2)
mini_portile (0.6.0)
multi_json (1.10.1)
nokogiri (1.6.2.1)
mini_portile (= 0.6.0)
nokogiri (1.6.2.1-java)
mini_portile (0.6.2)
multi_json (1.11.2)
nokogiri (1.6.6.2)
mini_portile (~> 0.6.0)
pry (0.9.12.2)
coderay (~> 1.0.5)
method_source (~> 0.8)
Expand All @@ -48,30 +48,29 @@ GEM
pry-nav (0.2.3)
pry (~> 0.9.10)
rake (10.1.1)
rspec (2.14.1)
rspec-core (~> 2.14.0)
rspec-expectations (~> 2.14.0)
rspec-mocks (~> 2.14.0)
rspec-core (2.14.5)
rspec-expectations (2.14.2)
diff-lcs (>= 1.1.3, < 2.0)
rspec-mocks (2.14.3)
rspec (3.1.0)
rspec-core (~> 3.1.0)
rspec-expectations (~> 3.1.0)
rspec-mocks (~> 3.1.0)
rspec-core (3.1.7)
rspec-support (~> 3.1.0)
rspec-expectations (3.1.2)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.1.0)
rspec-mocks (3.1.3)
rspec-support (~> 3.1.0)
rspec-support (3.1.2)
safe_yaml (1.0.3)
slop (3.4.6)
slyphon-log4j (1.2.15)
slyphon-zookeeper_jar (3.3.5-java)
spoon (0.0.4)
ffi
webmock (1.18.0)
addressable (>= 2.3.6)
crack (>= 0.3.2)
zk (1.9.4)
zk (1.9.5)
logging (~> 1.8.2)
zookeeper (~> 1.4.0)
zookeeper (1.4.8)
zookeeper (1.4.8-java)
slyphon-log4j (= 1.2.15)
slyphon-zookeeper_jar (= 3.3.5)
zookeeper (1.4.10)

PLATFORMS
java
Expand All @@ -81,6 +80,9 @@ DEPENDENCIES
pry
pry-nav
rake
rspec
rspec (~> 3.1.0)
synapse!
webmock

BUNDLED WITH
1.10.5
115 changes: 67 additions & 48 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ In an environment like Amazon's EC2, all of the available workarounds are subopt

* Round-robin DNS: Slow to converge, and doesn't work when applications cache DNS lookups (which is frequent)
* Elastic IPs: slow to converge, limited in number, public-facing-only, which makes them less useful for internal services
* ELB: Again, public-facing only, and only useful for HTTP
* ELB: ultimately uses DNS (see above), can't tune load balancing, have to launch a new one for every service * region, autoscaling doesn't happen fast enough

One solution to this problem is a discovery service, like [Apache Zookeeper](http://zookeeper.apache.org/).
However, Zookeeper and similar services have their own problems:
Expand Down Expand Up @@ -92,38 +92,50 @@ HAProxy will be transparently reloaded, and your application will keep running w

## Installation

Add this line to your application's Gemfile:
To download and run the synapse binary, first install a version of ruby. Then,
install synapse with:

gem 'synapse'

And then execute:
```bash
$ mkdir -p /opt/smartstack/synapse
# If you are on Ruby 2.X use --no-document instead of --no-ri --no-rdoc
$ gem install synapse --install-dir /opt/smartstack/synapse --no-ri --no-rdoc
```

$ bundle
This will download synapse and its dependencies into /opt/smartstack/synapse. You
might wish to omit the `--install-dir` flag to use your system's default gem
path, however this will require you to run `gem install synapse` with root
permissions.

Or install it yourself as:
You can now run the synapse binary like:

$ gem install synapse

```bash
export GEM_PATH=/opt/smartstack/synapse
/opt/smartstack/synapse/bin/synapse --help
```

Don't forget to install HAProxy prior to installing Synapse.
Don't forget to install HAProxy too.

## Configuration ##

Synapse depends on a single config file in JSON format; it's usually called `synapse.conf.json`.
The file has two main sections.
The first is the `services` section, which lists the services you'd like to connect.
The second is the `haproxy` section, which specifies how to configure and interact with HAProxy.
The file has three main sections.

1. [`services`](#services): lists the services you'd like to connect.
2. [`haproxy`](#haproxy): specifies how to configure and interact with HAProxy.
3. [`file_output`](#file) (optional): specifies where to write service state to on the filesystem.

<a name="services"/>
### Configuring a Service ###

The services are a hash, where the keys are the `name` of the service to be configured.
The `services` section is a hash, where the keys are the `name` of the service to be configured.
The name is just a human-readable string; it will be used in logs and notifications.
Each value in the services hash is also a hash, and should contain the following keys:

* `discovery`: how synapse will discover hosts providing this service (see below)
* [`discovery`](#discovery): how synapse will discover hosts providing this service (see below)
* `default_servers`: the list of default servers providing this service; synapse uses these if no others can be discovered
* `haproxy`: how will the haproxy section for this service be configured
* [`haproxy`](#haproxysvc): how will the haproxy section for this service be configured

<a name="discovery"/>
#### Service Discovery ####

We've included a number of `watchers` which provide service discovery.
Expand Down Expand Up @@ -209,6 +221,11 @@ If you do not list any default servers, no proxy will be created. The
`default_servers` will also be used in addition to discovered servers if the
`keep_default_servers` option is set.

If you do not list any `default_servers`, and all backends for a service
disappear then the previous known backends will be used. Disable this behavior
by unsetting `use_previous_backends`.

<a name="haproxysvc"/>
#### The `haproxy` Section ####

This section is its own hash, which should contain the following keys:
Expand All @@ -218,38 +235,62 @@ This section is its own hash, which should contain the following keys:
* `server_options`: the haproxy options for each `server` line of the service in HAProxy config; it may be left out.
* `frontend`: additional lines passed to the HAProxy config in the `frontend` stanza of this service
* `backend`: additional lines passed to the HAProxy config in the `backend` stanza of this service
* `backend_name`: The name of the generated HAProxy backend for this service
(defaults to the service's key in the `services` section)
* `listen`: these lines will be parsed and placed in the correct `frontend`/`backend` section as applicable; you can put lines which are the same for the frontend and backend here.
* `shared_frontend`: optional: haproxy configuration directives for a shared http frontend (see below)

<a name="haproxy"/>
### Configuring HAProxy ###

The `haproxy` section of the config file has the following options:
The top level `haproxy` section of the config file has the following options:

* `reload_command`: the command Synapse will run to reload HAProxy
* `config_file_path`: where Synapse will write the HAProxy config file
* `do_writes`: whether or not the config file will be written (default to `true`)
* `do_reloads`: whether or not Synapse will reload HAProxy (default to `true`)
* `do_socket`: whether or not Synapse will use the HAProxy socket commands to prevent reloads (default to `true`)
* `global`: options listed here will be written into the `global` section of the HAProxy config
* `defaults`: options listed here will be written into the `defaults` section of the HAProxy config
* `extra_sections`: additional, manually-configured `frontend`, `backend`, or `listen` stanzas
* `bind_address`: force HAProxy to listen on this address (default is localhost)
* `shared_fronted`: (OPTIONAL) additional lines passed to the HAProxy config used to configure a shared HTTP frontend (see below)
* `shared_frontend`: (OPTIONAL) additional lines passed to the HAProxy config used to configure a shared HTTP frontend (see below)
* `restart_interval`: number of seconds to wait between restarts of haproxy (default: 2)
* `restart_jitter`: percentage, expressed as a float, of jitter to multiply the `restart_interval` by when determining the next
restart time. Use this to help prevent healthcheck storms when HAProxy restarts. (default: 0.0)
* `state_file_path`: full path on disk (e.g. /tmp/synapse/state.json) for caching haproxy state between reloads.
If provided, synapse will store recently seen backends at this location and can "remember" backends across both synapse and
HAProxy restarts. Any backends that are "down" in the reporter but listed in the cache will be put into HAProxy disabled (default: nil)
* `state_file_ttl`: the number of seconds that backends should be kept in the state file cache.
This only applies if `state_file_path` is provided (default: 86400)

Note that a non-default `bind_address` can be dangerous.
If you configure an `address:port` combination that is already in use on the system, haproxy will fail to start.

<a name="file"/>
### Configuring `file_output` ###

This section controls whether or not synapse will write out service state
to the filesystem in json format. This can be used for services that want to
use discovery information but not go through HAProxy.

* `output_directory`: the path to a directory on disk that service registrations
should be written to.


### HAProxy shared HTTP Frontend ###

For HTTP-only services, it is not always necessary or desirable to dedicate a TCP port per service, since HAProxy can route traffic based on host headers.
To support this, the optional `shared_fronted` section can be added to both the `haproxy` section and each indvidual service definition.
To support this, the optional `shared_frontend` section can be added to both the `haproxy` section and each indvidual service definition.
Synapse will concatenate them all into a single frontend section in the generated haproxy.cfg file.
Note that synapse does not assemble the routing ACLs for you; you have to do that yourself based on your needs.
This is probably most useful in combination with the `service_conf_dir` directive in a case where the individual service config files are being distributed by a configuration manager such as puppet or chef, or bundled into service packages.
For example:

```yaml
haproxy:
shared_frontend: "bind 127.0.0.1:8081"
shared_frontend:
- "bind 127.0.0.1:8081"
reload_command: "service haproxy reload"
config_file_path: "/etc/haproxy/haproxy.cfg"
socket_file_path: "/var/run/haproxy.sock"
Expand All @@ -268,7 +309,8 @@ For example:
discovery:
method: "zookeeper"
path: "/nerve/services/service1"
hosts: "0.zookeeper.example.com:2181"
hosts:
- "0.zookeeper.example.com:2181"
haproxy:
server_options: "check inter 2s rise 3 fall 2"
shared_frontend:
Expand All @@ -287,7 +329,8 @@ For example:
shared_frontend:
- "acl is_service1 hdr_dom(host) -i service2.lb.example.com"
- "use_backend service2 if is_service2
backend: "mode http"
backend:
- "mode http"
```

Expand Down Expand Up @@ -322,29 +365,5 @@ Non-HTTP backends such as MySQL or RabbitMQ will obviously continue to need thei

### Creating a Service Watcher ###

If you'd like to create a new service watcher:

1. Create a file for your watcher in `service_watcher` dir
2. Use the following template:
```ruby
require 'synapse/service_watcher/base'
module Synapse
class NewWatcher < BaseWatcher
def start
# write code which begins running service discovery
end
private
def validate_discovery_opts
# here, validate any required options in @discovery
end
end
end
```

3. Implement the `start` and `validate_discovery_opts` methods
4. Implement whatever additional methods your discovery requires

When your watcher detects a list of new backends, they should be written to `@backends`.
You should then call `@synapse.configure` to force synapse to update the HAProxy config.
See the Service Watcher [README](lib/synapse/service_watcher/README.md) for
how to add new Service Watchers.
38 changes: 26 additions & 12 deletions lib/synapse.rb
View file
Original file line number Diff line number Diff line change
@@ -1,25 +1,34 @@
require 'logger'
require 'json'

require "synapse/version"
require "synapse/service_watcher/base"
require "synapse/log"
require "synapse/haproxy"
require "synapse/file_output"
require "synapse/service_watcher"
require "synapse/log"

require 'logger'
require 'json'

include Synapse

module Synapse
class Synapse

include Logging

def initialize(opts={})
# create the service watchers for all our services
raise "specify a list of services to connect in the config" unless opts.has_key?('services')
@service_watchers = create_service_watchers(opts['services'])

# create the haproxy object
# create objects that need to be notified of service changes
@config_generators = []
# create the haproxy config generator, this is mandatory
raise "haproxy config section is missing" unless opts.has_key?('haproxy')
@haproxy = Haproxy.new(opts['haproxy'])
@config_generators << Haproxy.new(opts['haproxy'])

# possibly create a file manifestation for services that do not
# want to communicate via haproxy, e.g. cassandra
if opts.has_key?('file_output')
@config_generators << FileOutput.new(opts['file_output'])
end

# configuration is initially enabled to configure on first loop
@config_updated = true
Expand Down Expand Up @@ -47,10 +56,15 @@ def run

if @config_updated
@config_updated = false
log.info "synapse: regenerating haproxy config"
@haproxy.update_config(@service_watchers)
else
sleep 1
@config_generators.each do |config_generator|
log.info "synapse: configuring #{config_generator.name}"
config_generator.update_config(@service_watchers)
end
end

sleep 1
@config_generators.each do |config_generator|
config_generator.tick(@service_watchers)
end

loops += 1
Expand Down
Loading

0 comments on commit aa4cbc5

Please sign in to comment.