Skip to content

Commit

Permalink
rename Poller to Toiler
Browse files Browse the repository at this point in the history
  • Loading branch information
sschepens committed Apr 29, 2015
1 parent a429173 commit cb678b4
Show file tree
Hide file tree
Showing 19 changed files with 143 additions and 143 deletions.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Copyright (c) Sebastian Schepens

Poller is an Open Source project licensed under the terms of
Toiler is an Open Source project licensed under the terms of
the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
for license text.

40 changes: 20 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
##Poller
Poller is a AWS SQS long-polling thread-based message processor.
It's based on [poller](https://github.com/phstc/poller) but takes
##Toiler
Toiler is a AWS SQS long-polling thread-based message processor.
It's based on [shoryuken](https://github.com/phstc/shoryuken) but takes
a different approach at loadbalancing and uses long-polling.

##Features
###Concurrency
Poller allows to specify the amount of processors (threads) that should be spawned for each queue.
Instead of [poller's](https://github.com/phstc/poller) loadbalancing approach, Poller delegates this work to the kernel scheduling threads.
Toiler allows to specify the amount of processors (threads) that should be spawned for each queue.
Instead of [shoryuken's](https://github.com/phstc/shoryuken) loadbalancing approach, Toiler delegates this work to the kernel scheduling threads.

###Long-Polling
A Fetcher thread is spawned for each queue.
Expand All @@ -18,14 +18,14 @@ By long-polling fetchers wait for a configurable amount of time for messages to
Workers can configure a parser Class or Proc to parse an SQS message body before being processed.

###Batches
Poller allows a Worker to be able to receive a batch of messages instead of a single one.
Toiler allows a Worker to be able to receive a batch of messages instead of a single one.

##Instalation

Add this line to your application's Gemfile:

```ruby
gem 'poller'
gem 'toiler'
```

And then execute:
Expand All @@ -34,23 +34,23 @@ And then execute:

Or install it yourself as:

$ gem install poller
$ gem install toiler

## Usage

### Worker class

```ruby
class MyWorker
include Poller::Worker
include Toiler::Worker

poller_options queue: 'default', concurrency: 5, auto_delete: true
poller_options parser: :json
toiler_options queue: 'default', concurrency: 5, auto_delete: true
toiler_options parser: :json

# poller_options parser: ->(sqs_msg){ REXML::Document.new(sqs_msg.body) }
# poller_options parser: MultiJson
# poller_options auto_visibility_timeout: true
# poller_options batch: true
# toiler_options parser: ->(sqs_msg){ REXML::Document.new(sqs_msg.body) }
# toiler_options parser: MultiJson
# toiler_options auto_visibility_timeout: true
# toiler_options batch: true

def perform(sqs_msg, body)
puts body
Expand All @@ -70,21 +70,21 @@ wait: 20 # The time in seconds to wait for messages durin
### Rails Integration
You can tell Poller to load your Rails application by passing the `-R` or `--rails` flag to the "poller" command.
You can tell Toiler to load your Rails application by passing the `-R` or `--rails` flag to the "toiler" command.

If you load Rails, and assuming your workers are located in the `app/workers` directory, they will be auto-loaded. This means you don't need to require them explicitly with `-r`.


### Start Poller
### Start Toiler

```shell
bundle exec poller -r worker.rb -C poller.yml
bundle exec toiler -r worker.rb -C toiler.yml
```

Other options:

```bash
poller --help
toiler --help
-d, --daemon Daemonize process
-r, --require [PATH|DIR] Location of the worker
Expand All @@ -103,7 +103,7 @@ Much of the credit goes to [Pablo Cantero](https://github.com/phstc), creator of

## Contributing

1. Fork it ( https://github.com/sschepens/poller/fork )
1. Fork it ( https://github.com/sschepens/toiler/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
Expand Down
4 changes: 2 additions & 2 deletions bin/poller → bin/toiler
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/usr/bin/env ruby

require 'poller'
require 'toiler'

begin
Poller::CLI.instance.run(ARGV)
Toiler::CLI.instance.run(ARGV)
rescue => e
raise e if $DEBUG
STDERR.puts e.message
Expand Down
42 changes: 0 additions & 42 deletions lib/poller/worker.rb

This file was deleted.

22 changes: 11 additions & 11 deletions lib/poller.rb → lib/toiler.rb
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
require 'aws-sdk'
require 'poller/core_ext'
require 'poller/message'
require 'poller/queue'
require 'poller/worker'
require 'poller/environment_loader'
require 'poller/logging'
require 'poller/cli'
require 'poller/version'

module Poller
require 'toiler/core_ext'
require 'toiler/message'
require 'toiler/queue'
require 'toiler/worker'
require 'toiler/environment_loader'
require 'toiler/logging'
require 'toiler/cli'
require 'toiler/version'

module Toiler
@worker_registry = {}
@worker_class_registry = {}
@options = {
Expand All @@ -22,7 +22,7 @@ def options
end

def logger
Poller::Logging.logger
Toiler::Logging.logger
end

def worker_class_registry
Expand Down
22 changes: 11 additions & 11 deletions lib/poller/cli.rb → lib/toiler/cli.rb
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
require 'singleton'
require 'optparse'
require 'poller'
require 'toiler'

module Poller
module Toiler
# See: https://github.com/mperham/sidekiq/blob/33f5d6b2b6c0dfaab11e5d39688cab7ebadc83ae/lib/sidekiq/cli.rb#L20
class Shutdown < Interrupt; end

Expand Down Expand Up @@ -49,20 +49,20 @@ def handle_signal(_signal)
end

def load_celluloid
fail "Celluloid cannot be required until here, or it will break Poller's daemonization" if defined?(::Celluloid) && Poller.options[:daemon]
fail "Celluloid cannot be required until here, or it will break Toiler's daemonization" if defined?(::Celluloid) && Toiler.options[:daemon]

# Celluloid can't be loaded until after we've daemonized
# because it spins up threads and creates locks which get
# into a very bad state if forked.
require 'celluloid/autostart'
Celluloid.logger = (Poller.options[:verbose] ? Poller.logger : nil)
require 'poller/supervisor'
Celluloid.logger = (Toiler.options[:verbose] ? Toiler.logger : nil)
require 'toiler/supervisor'
end

def daemonize
return unless Poller.options[:daemon]
return unless Toiler.options[:daemon]

fail ArgumentError, "You really should set a logfile if you're going to daemonize" unless Poller.options[:logfile]
fail ArgumentError, "You really should set a logfile if you're going to daemonize" unless Toiler.options[:logfile]

files_to_reopen = []
ObjectSpace.each_object(File) do |file|
Expand All @@ -80,7 +80,7 @@ def daemonize
end

[$stdout, $stderr].each do |io|
File.open(Poller.options[:logfile], 'ab') do |f|
File.open(Toiler.options[:logfile], 'ab') do |f|
io.reopen(f)
end
#io.sync = true
Expand All @@ -89,7 +89,7 @@ def daemonize
end

def write_pid
if (path = Poller.options[:pidfile])
if (path = Toiler.options[:pidfile])
File.open(path, 'w') do |f|
f.puts Process.pid
end
Expand Down Expand Up @@ -129,9 +129,9 @@ def parse_cli_args(argv)
end
end

@parser.banner = 'poller [options]'
@parser.banner = 'toiler [options]'
@parser.on_tail '-h', '--help', 'Show help' do
Poller.logger.info @parser
Toiler.logger.info @parser
exit 1
end
@parser.parse!(argv)
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
require 'erb'
require 'yaml'

module Poller
module Toiler
class EnvironmentLoader
attr_reader :options

Expand All @@ -10,7 +10,7 @@ def self.load(options)
end

def self.load_for_rails_console
load(config_file: (Rails.root + 'config' + 'poller.yml'))
load(config_file: (Rails.root + 'config' + 'toiler.yml'))
end

def initialize(options)
Expand All @@ -21,8 +21,8 @@ def load
initialize_logger
load_rails if options[:rails]
require_workers if options[:require]
Poller.options.merge!(config_file_options)
Poller.options.merge!(options)
Toiler.options.merge!(config_file_options)
Toiler.options.merge!(options)
initialize_aws
end

Expand All @@ -31,7 +31,7 @@ def load
def config_file_options
if (path = options[:config_file])
unless File.exist?(path)
Poller.logger.warn "Config file #{path} does not exist"
Toiler.logger.warn "Config file #{path} does not exist"
path = nil
end
end
Expand All @@ -44,16 +44,16 @@ def config_file_options
def initialize_aws
# aws-sdk tries to load the credentials from the ENV variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
# when not explicit supplied
fail 'AWS Credentials needed!' if Poller.options[:aws].empty? && (ENV['AWS_ACCESS_KEY_ID'].nil? || ENV['AWS_SECRET_ACCESS_KEY'].nil?)
return if Poller.options[:aws].empty?
fail 'AWS Credentials needed!' if Toiler.options[:aws].empty? && (ENV['AWS_ACCESS_KEY_ID'].nil? || ENV['AWS_SECRET_ACCESS_KEY'].nil?)
return if Toiler.options[:aws].empty?

::Aws.config[:region] = Poller.options[:aws][:region]
::Aws.config[:credentials] = ::Aws::Credentials.new Poller.options[:aws][:access_key_id], Poller.options[:aws][:secret_access_key]
::Aws.config[:region] = Toiler.options[:aws][:region]
::Aws.config[:credentials] = ::Aws::Credentials.new Toiler.options[:aws][:access_key_id], Toiler.options[:aws][:secret_access_key]
end

def initialize_logger
Poller::Logging.initialize_logger(options[:logfile]) if options[:logfile]
Poller.logger.level = Logger::DEBUG if options[:verbose]
Toiler::Logging.initialize_logger(options[:logfile]) if options[:logfile]
Toiler.logger.level = Logger::DEBUG if options[:verbose]
end

def load_rails
Expand All @@ -66,13 +66,13 @@ def load_rails
else
# Painful contortions, see 1791 for discussion
require File.expand_path('config/application.rb')
::Rails::Application.initializer 'poller.eager_load' do
::Rails::Application.initializer 'toiler.eager_load' do
::Rails.application.config.eager_load = true
end
require File.expand_path('config/environment.rb')
end

Poller.logger.info 'Rails environment loaded'
Toiler.logger.info 'Rails environment loaded'
end

def require_workers
Expand Down
12 changes: 6 additions & 6 deletions lib/poller/fetcher.rb → lib/toiler/fetcher.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module Poller
module Toiler
class Fetcher
include Celluloid
include Celluloid::Logger
Expand All @@ -11,8 +11,8 @@ class Fetcher

def initialize(queue, client = nil)
@queue = Queue.new queue, client
@wait = Poller.options[:wait] || 20
@batch = Poller.worker_class_registry[queue].batch?
@wait = Toiler.options[:wait] || 20
@batch = Toiler.worker_class_registry[queue].batch?
async.poll_messages
end

Expand All @@ -28,11 +28,11 @@ def poll_messages
}

loop do
count = Poller.manager.free_processors queue.name
count = Toiler.manager.free_processors queue.name
options[:max_number_of_messages] = (batch || count > FETCH_LIMIT) ? FETCH_LIMIT : count
msgs = queue.receive_messages options
Poller.manager.assign_messages queue.name, msgs unless msgs.empty?
Poller.manager.wait_for_available_processors queue.name
Toiler.manager.assign_messages queue.name, msgs unless msgs.empty?
Toiler.manager.wait_for_available_processors queue.name
end
end
end
Expand Down
Loading

0 comments on commit cb678b4

Please sign in to comment.