Skip to content

Commit

Permalink
Merge pull request #18 from Ilhasoft/update/v7.4.0
Browse files Browse the repository at this point in the history
Update/v7.4.0
  • Loading branch information
paulobernardoaf authored May 4, 2023
2 parents f8918a1 + 5c7bf27 commit bd8f651
Show file tree
Hide file tree
Showing 22 changed files with 1,611 additions and 1,269 deletions.
84 changes: 42 additions & 42 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,65 +1,65 @@
name: CI
on: [push, pull_request]
env:
go-version: '1.16.x'
es-version: '7.10.1'
go-version: "1.18.x"
es-version: "7.17.2"
jobs:
test:
name: Test
strategy:
matrix:
pg-version: ['12', '13']
pg-version: ["12", "13"]
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v1
- name: Checkout code
uses: actions/checkout@v1

- name: Install ElasticSearch
uses: nyaruka/elasticsearch-action@master
with:
elastic version: ${{ env.es-version }}
- name: Install ElasticSearch
uses: nyaruka/elasticsearch-action@master
with:
elastic version: ${{ env.es-version }}

- name: Install PostgreSQL
uses: harmon758/postgresql-action@v1
with:
postgresql version: ${{ matrix.pg-version }}
postgresql db: elastic_test
postgresql user: temba
postgresql password: temba
- name: Install PostgreSQL
uses: harmon758/postgresql-action@v1
with:
postgresql version: ${{ matrix.pg-version }}
postgresql db: elastic_test
postgresql user: nyaruka
postgresql password: nyaruka

- name: Install Go
uses: actions/setup-go@v1
with:
go-version: ${{ env.go-version }}
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: ${{ env.go-version }}

- name: Run tests
run: go test -p=1 -coverprofile=coverage.text -covermode=atomic ./...
- name: Run tests
run: go test -p=1 -coverprofile=coverage.text -covermode=atomic ./...

- name: Upload coverage
if: success()
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}

- name: Upload coverage
if: success()
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}

release:
name: Release
needs: [test]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v1
- name: Checkout code
uses: actions/checkout@v1

- name: Install Go
uses: actions/setup-go@v1
with:
go-version: ${{ env.go-version }}
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: ${{ env.go-version }}

- name: Publish release
uses: goreleaser/goreleaser-action@v1
with:
version: v0.147.2
args: release --rm-dist
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
fail_ci_if_error: true
- name: Publish release
uses: goreleaser/goreleaser-action@v1
with:
version: v0.147.2
args: release --rm-dist
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
fail_ci_if_error: true
4 changes: 1 addition & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,7 @@
*.dylib
fabfile.py
fabfile.pyc
fabconfig.py
fabconfig.pyc
fabric
deploy/
rp-indexer

# Test binary, build with `go test -c`
Expand Down
61 changes: 61 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,64 @@
v7.4.0
----------
* Update README
* Tweak startup logging

v7.3.10
----------
* Log app version on startup

v7.3.9
----------
* Use analytics package from gocommon instead of librato directly
* Add arm64 as a build target

v7.3.8
----------
* Update dependencies and go version to 1.18
* Don't panic on connection failure to ES

v7.3.7
----------
* Better logging within batches during rebuilds
* Test with latest ES 7.17

v7.3.6
----------
* Ignore malformed field value numbers
* Drop the flow and groups fields which have been replaced by flow_id and group_ids

v7.3.5
----------
* Log batch progress during rebuilds

v7.3.4
----------
* Add group_ids field to replace groups

v7.3.3
----------
* Include flow id history as flow_history_ids and current flow id as flow_id

v7.3.1
----------
* If indexing fails, log status code from elasticsearch
* Poll interval is configurable

v7.3.0
----------
* Add stats reporting cron task and optional librato config
* Refactor to support different indexer types
* Update golang.org/x/sys

v7.2.0
----------
* Tweak README

v7.1.0
----------
* Index contact.current_flow_id as flow uuid
* CI with go 1.17

v7.0.0
----------
* Test on PG12 and 13
Expand Down
102 changes: 49 additions & 53 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,89 @@
# RapidPro Indexer
# Indexer

[![Build Status](https://github.com/nyaruka/rp-indexer/workflows/CI/badge.svg)](https://github.com/nyaruka/rp-indexer/actions?query=workflow%3ACI)
[![codecov](https://codecov.io/gh/nyaruka/rp-indexer/branch/main/graph/badge.svg)](https://codecov.io/gh/nyaruka/rp-indexer)
[![Go Report Card](https://goreportcard.com/badge/github.com/nyaruka/rp-indexer)](https://goreportcard.com/report/github.com/nyaruka/rp-indexer)

Simple service for indexing RapidPro contacts into ElasticSearch.
Service for indexing RapidPro/TextIt contacts into Elasticsearch.

# Deploying
## Deploying

As Indexer is a Go application, it compiles to a binary and that binary along with the config file is all
As it is a Go application, it compiles to a binary and that binary along with the config file is all
you need to run it on your server. You can find bundles for each platform in the
[releases directory](https://github.com/nyaruka/rp-indexer/releases). You should only run a single indexer
[releases directory](https://github.com/nyaruka/rp-indexer/releases). You should only run a single
instance for a deployment.

Indexer can run in two modes:
It can run in two modes:

1) the default mode, which simply queries the ElasticSearch database, finds the most recently
modified contact, then on a schedule queries the `contacts_contact` table on the RapidPro
modified contact, then on a schedule queries the `contacts_contact` table in the
database for contacts to add or delete. You should run this as a long running service which
constantly keeps ElasticSearch in sync with your RapidPro contacts.
constantly keeps ElasticSearch in sync with your contacts.

2) a rebuild mode, started with `--rebuild`. This builds a brand new index from nothing, querying
all contacts on RapidPro. Once complete, this switches out the alias for the contact index
with the newly build index. This can be run on a cron (in parallel with the mode above) to rebuild
your index occasionally to get rid of bloat.

# Configuration
## Configuration

The service uses a tiered configuration system, each option takes precendence over the ones above it:

Indexer uses a tiered configuration system, each option takes precendence over the ones above it:
1. The configuration file
2. Environment variables starting with `INDEXER_`
3. Command line parameters

We recommend running Indexer with no changes to the configuration and no parameters, using only
We recommend running it with no changes to the configuration and no parameters, using only
environment variables to configure it. You can use `% rp-indexer --help` to see a list of the
environment variables and parameters and for more details on each option.

## RapidPro Configuration
### RapidPro

For use with RapidPro, you will want to configure these settings:

* `INDEXER_DB`: a URL connection string for your RapidPro database
* `INDEXER_DB`: a URL connection string for your RapidPro database or read replica
* `INDEXER_ELASTIC_URL`: the URL for your ElasticSearch endpoint

Recommended settings for error reporting:

* `INDEXER_SENTRY_DSN`: The DSN to use when logging errors to Sentry

# Development
### Reference

These are the configuration options that can be provided as parameters or environment variables. If using environment
varibles, convert to uppercase, replace dashes with underscores and prefix the name with `INDEXER_`, e.g. `-log-level`
becomes `INDEXER_LOG_LEVEL`.

```
-cleanup
whether to remove old indexes after a rebuild
-db string
the connection string for our database (default "postgres://localhost/rapidpro?sslmode=disable")
-debug-conf
print where config values are coming from
-elastic-url string
the url for our elastic search instance (default "http://localhost:9200")
-help
print usage information
-index string
the alias for our contact index (default "contacts")
-librato-username
the Librato username for metrics reporting
-librato-token
the Librato token for metrics reporting
-log-level string
the log level, one of error, warn, info, debug (default "info")
-poll int
the number of seconds to wait between checking for updated contacts (default 5)
-rebuild
whether to rebuild the index, swapping it when complete, then exiting (default false)
-sentry-dsn string
the sentry configuration to log errors to, if any
```

## Development

Once you've checked out the code, you can build Indexer with:
Once you've checked out the code, you can build the service with:

```
go build github.com/nyaruka/rp-indexer/cmd/rp-indexer
Expand All @@ -68,41 +102,3 @@ To run all of the tests:
```
go test ./... -p=1
```

# Usage

```
Indexes RapidPro contacts to ElasticSearch
Usage of indexer:
-cleanup
whether to remove old indexes after a rebuild
-db string
the connection string for our database (default "postgres://localhost/rapidpro?sslmode=disable")
-debug-conf
print where config values are coming from
-elastic-url string
the url for our elastic search instance (default "http://localhost:9200")
-help
print usage information
-index string
the alias for our contact index (default "contacts")
-log-level string
the log level, one of error, warn, info, debug (default "info")
-poll int
the number of seconds to wait between checking for updated contacts (default 5)
-rebuild
whether to rebuild the index, swapping it when complete, then exiting (default false)
-sentry-dsn string
the sentry configuration to log errors to, if any
Environment variables:
INDEXER_CLEANUP - bool
INDEXER_DB - string
INDEXER_ELASTIC_URL - string
INDEXER_INDEX - string
INDEXER_LOG_LEVEL - string
INDEXER_POLL - int
INDEXER_REBUILD - bool
INDEXER_SENTRY_DSN - string
```
Loading

0 comments on commit bd8f651

Please sign in to comment.