Skip to content

dtmirizzi/target-elasticsearch

Repository files navigation

target-elasticsearch

target-elasticsearch is a Singer target for Elastic.

Some target development because I am currently hacking together something together with a bash script Built with the Meltano Tap SDK for Singer Taps.

Video

video

Settings

Setting Required Default Description
scheme True http http scheme used for connecting to elasticsearch
host True localhost host used to connect to elasticsearch
port True 9200 port use to connect to elasticsearch
username False None basic auth username as per elastic documentation
password False None basic auth password as per elastic documentation
bearer_token False None Bearer token for bearer authorization as per elastic documentation
api_key_id False None api key id for auth key authorization as per elastic documentation
api_key False None api key for auth key authorization as per elastic documentation
ssl_ca_file False None location of the the SSL certificate for cert verification ie. /some/path as per elastic documentation
index_format False ecs-{{ stream_name }}-{{ current_timestamp_daily }} can be used to handle custom index formatting such as specifying -latest index. Default options: Daily {{ current_timestamp_daily }}, Monthly {{ current_timestamp_monthly }}, or Yearly {{ current_timestamp_yearly }}. You should use fields specified in index_schema_fields such as {{ _id }} or {{ timestamp }} . There are also helper fuctions such as {{ to_daily(timestamp) }}`.
index_schema_fields False None this id map allows you to specify specific record values via jsonpath from the stream to be used in index formulation.
metadata_fields false None this should be used to pull out specific fields via jsonpath to be used on for ecs metadata patters
request_timeout false 10 increase timeout to send big butches of data Elasticsearch connection arguments
retry_on_timeout false True increase timeout to send big butches of data Elasticsearch connection arguments

A full list of supported settings and capabilities is available by running: target-elasticsearch --about

Installation

git clone [email protected]:dtmirizzi/target_elasticsearch.git
cd target_elasticsearch
pipx install .

Configuration

Accepted Config Options

A full list of supported settings and capabilities for this tap is available by running:

target_elasticsearch --about

Targey Authentication and Authorization

You can easily run target-elasticsearch by itself or in a pipeline using Meltano.

There are 4 types of auth supported by this target:

  • Basic Auth
  • Access Token
  • Bearer Token
  • SSL Based

Executing the Tap Directly

target_elasticsearch --version
target_elasticsearch --help
target_elasticsearch --config CONFIG --discover > ./catalog.json

Developer Resources

Initialize your Development Environment

pipx install poetry
poetry install
poetry run pre-commit install

Create and Run Tests

  1. Create tests within the target_elasticsearch/tests subfolder
  2. To Run tests, first set environment variables in your current shell for all required settings from the Settings section above. Example for UNIX base systems running export Target_elasticsearch_USER_ID="xxxxx" then run:
poetry run pytest

You can also test the target-elasticsearch CLI interface directly using poetry run:

poetry run target_elasticsearch --help

Testing with Meltano

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Your project comes with a custom meltano.yml project file already created.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd target-elasticsearch
meltano install
# You can spin up elastic locally
make local-es

Now you can test and orchestrate using Meltano:

meltano install
# Test invocation:
meltano invoke target-elasticsearch --version
# test configuration
meltano config target-elasticsearch set --interactive
# OR run a test `elt` pipeline:
meltano elt tap-smoke-test target-elasticsearch

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.