-
Notifications
You must be signed in to change notification settings - Fork 4
Backends
Backends are the core of neoq. Anything that implements the neoq.Neoq interface is a "backend".
Neoq provides three native backends: memory, postgres, and redis. However, users may implement neoq.Neoq to create their own backends for their preferred queue technology. Contributors to the neoq project are encouraged to open issues if they'd like to discuss adding any additional backends.
Each backend provides the same core functionality consisting of:
- Jobs may be added to arbitrary queues to be processed by Handlers
- Job Handler errors are retried with exponential backoff
- Job Timeouts and Deadlines must be enforced
- Concurrency per Handler/Queue must be configurable
- Periodic jobs can be registered to run as frequently as every second
The memory backend is intended for jobs that do not require any queue durabuility. If neoq is shut down or restarted, all in-memory queue contents are lost.
Intended use cases
- Testing and development
- Application periodic/cron jobs
- Sending time-sensitive notifications to users with short Deadlines
- Use your imagination; these are but a few examples
The in-memory backend has no additional configuration options.
The Postgres backend is a great option for applications that already use Postgres as a data store. Because Postgres is a general purpose RDBMs, enqueing jobs on the Postgres backend may be much slower than both the Redis and Memory backends. However, if your application queues fewer than dozens of jobs per second, this is likely a viable option for you.
Neoq is happy to use your exising application's database to store jobs, but if you have siginificant scaling needs, consider using a separate database and/or database server.
Neoq creates all the tables necessary for it to start processing jobs. As long as the credentials provided to neoq have table creation access, there is no need to create any tables in advance.
TODO: Document how to use neoq when the authenticating user does not have access to create tables.
Description
WithConnectionString declares the connection string to be used when connecting to the postgres server.
Example
neoq.New(ctx,
neoq.WithBackend(postgres.Backend),
postgres.WithConnectionString("postgres://postgres:[email protected]:5432/neoq?sslmode=disable"),
)WithTransactionTimeout delcares the amount of time (ms) transactions may sit idle before being cancelled and postgres drops the transaction's underying connection. Neoq does not hold transactions while executing jobs, so the transaction timeout may be lower than the duration of your jobs.
Neoq begins transactions in the following scenarios: enqueueing jobs, notifying workers of new jobs, and updating job status.
Example
Connect to Postgres at 127.0.0.1 on port 5432 with username postgres and password postgres, storing jobs in database neoq, with SSL disabled.
nq, err := neoq.New(ctx,
neoq.WithBackend(postgres.Backend),
postgres.WithConnectionString("postgres://postgres:[email protected]:5432/neoq?sslmode=disable"),
postgres.WithTransactionTimeout(20000), // set the transaction timeout to 20 seconds
)Neoq's Redis backend is unique in that it uses the wonderful github.com/hibiken/asynq library under the hood, rather than being a first-party Redis implmenetation.
The decision to use an existing job processor in lieu of a first-party implementation was motivated by Asynq's rich feature set, good performance, and stable API. If a concrete case can be made as to why Asynq is not a good choice, please open an issue to start a discussion.
WithAddr sets the Redis server address (addr) to connect to. The address must be of the form: <HOST>:<PORT>.
Example
Connect to the Redis server at localhost on port 6379 without a password.
nq, _ := neoq.New(ctx,
neoq.WithBackend(redis.Backend),
redis.WithAddr("localhost:6379"),
redis.WithPassword(""),
)WithPassword sets the Redis password with which to authenticate
Example
Connect to the Redis server at localhost on port 6379 with password foobar.
nq, _ := neoq.New(ctx,
neoq.WithBackend(redis.Backend),
redis.WithAddr("localhost:6379"),
redis.WithPassword("foobar"),
)With concurrency configures the number of workers available to process jobs across all queues.
Example
Allow up to 2 concurrent jobs to be processed across all queues.
nq, _ := neoq.New(ctx,
neoq.WithBackend(redis.Backend),
redis.WithAddr("localhost:6379"),
redis.WithPassword("foobar"),
redis.WithConcurrency(2),
)WithShutdownTimeout specifies the duration to wait to let workers finish their tasks before forcing them to abort during neoq.Shutdown(). If unset or zero, default timeout of 8 seconds is used.
This duration should be at least as long as your longest running job, to allow an existing jobs to complete during shutdown.
Example
Allow a 10 second grace period for jobs to complete during shutdown.
nq, err := neoq.New(
ctx,
neoq.WithBackend(Backend),
redis.WithAddr("localhost:6379"),
redis.WithPassword("foobar"),
redis.WithShutdownTimeout(10 * time.Second),
)