Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/reference/connecting-to-serverless.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Set the value to port :443 instead.

::::

## Communication between {{ls}} {{es-serverless}} [connecting-to-elasticsearch-serverless]
## Communication between {{ls}} and {{es-serverless}} [connecting-to-elasticsearch-serverless]

[{{es-serverless}}](docs-content://solutions/search/serverless-elasticsearch-get-started.md) simplifies safe, secure communication between {{ls}} and {{es}}.
When you configure the Elasticsearch output plugin to use [`cloud_id`](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-cloud_id) and an [`api_key`](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-api_key), no additional SSL configuration is needed.
Expand Down
41 changes: 26 additions & 15 deletions docs/reference/ea-integration-tutorial.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,34 @@
# Tutorial: Using the {{ls}} `elastic_integration` filter to extend Elastic {{integrations}} [ea-integrations-tutorial]

You can use {{ls}} to transform events collected by {{agent}} and paired with an [Elastic integration](integration-docs://reference/index.md).
---
navigation_title: Tutorial to extend Elastic Integrations
applies_to:
stack: ga
serverless: ga
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added references to Serverless throughout as a reminder to see if that's a viable option for this tutorial.

products:
- id: elasticsearch
- id: cloud-hosted
- id: cloud-enterprise
- id: cloud-kubernetes
---

# Tutorial: Extend Elastic {{integrations}} with the {{ls}} elastic_integration filter[ea-integrations-tutorial]

You can use {{ls}} to transform events collected by {{agent}}s and paired with an [Elastic integration](integration-docs://reference/index.md).

You get the benefits of Elastic integrations--such as the simplicity of ingesting data from a wide variety of data sources and ensuring compliance with the [Elastic Common Schema (ECS)](ecs://reference/index.md)--combined with the extra processing power of {{ls}}.

This new functionality is made possible by the [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) plugin.
This new functionality is made possible by the [elastic_integration filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) plugin.
When you include the `elastic_integration` filter in your configuration, {{ls}} reads certain field values generated by the {{agent}},
and uses them to apply the transformations from Elastic integrations.
This allows you to further process events in the Logstash pipeline before sending them to their configured destinations.
This ability allows you to further process events in the Logstash pipeline before sending them to their configured destinations.

This tutorial walks you through adding the [Crowdstrike integration](integration-docs://reference/crowdstrike-intro.md) sending the data to {{ech}} or self-managed {{es}}.
This tutorial walks you through adding the [Crowdstrike integration](integration-docs://reference/crowdstrike.md), and sending the data to {{ech}}, {{serverless-full}}, or self-managed {{es}}.


## Prerequisites [ea-integrations-prereqs]

You need:

* A working {{es}} cluster
* A working {{ech}} or {{serverless-full}} deployment, or self-managed {{es}} cluster
* A {{ls}} instance
* {{fleet-server}}
* An [{{agent}} installed](docs-content://reference/fleet/install-elastic-agents.md) on the hosts you want to collect data from, and configured to [send output to {{ls}}](docs-content://reference/fleet/logstash-output.md)
Expand Down Expand Up @@ -70,10 +82,9 @@ This policy should be selected by default.

1. Create a new [{{ls}} pipeline](/reference/creating-logstash-pipeline.md).
2. Be sure to include these plugins:

* [`elastic_agent` input](logstash-docs-md://lsr/plugins-inputs-elastic_agent.md)
* [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md)
* [`elasticsearch` output](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md)
* [elastic_agent input](logstash-docs-md://lsr/plugins-inputs-elastic_agent.md)
* [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md)
* [`elasticsearch` output](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice to include input/output! ❤️


Note that every event sent from the {{agent}} to {{ls}} contains specific meta-fields.
{{ls}} expects events to contain a top-level `data_stream` field with `type`, `dataset`, and `namespace` sub-fields.
Expand All @@ -82,9 +93,9 @@ Note that every event sent from the {{agent}} to {{ls}} contains specific meta-f
{{ls}} frequently synchronizes with {{es}} to ensure that it has the most recent versions of the enabled integrations.


### Sample configuration: output to {{ech}} [ea-integrations-ess-sample]
### Sample configuration: output to Elastic Cloud [ea-integrations-ech-sample]

This sample illustrates using the `elastic_agent` input and the `elastic_integration` filter for processing in {{ls}}, and then sending the output to {{ech}}.
This sample illustrates using the `elastic_agent` input and the `elastic_integration` filter for processing in {{ls}}, and then sending the output to {{ech}} or {{serverless-full}}.

Check out the [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) for the full list of configuration options.

Expand All @@ -109,7 +120,7 @@ output {
}
```

All processing occurs in {{ls}} before events are forwarded to {{ech}}.
All processing occurs in {{ls}} before events are forwarded to {{ech}} or {{serverless-full}}.


### Sample configuration: output to self-managed {{es}} [ea-integrations-es-sample]
Expand Down Expand Up @@ -141,7 +152,7 @@ output {
}
## add elasticsearch
elasticsearch {
hosts => "{es-host}:9200"
hosts => ["{es-host}:9200"]
user => "elastic"
password => "changeme"
ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem"
Expand Down
11 changes: 11 additions & 0 deletions docs/reference/using-logstash-with-elastic-integrations.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,16 @@
---

Check notice on line 1 in docs/reference/using-logstash-with-elastic-integrations.md

View workflow job for this annotation

GitHub Actions / docs-preview / build

Irregular space detected. Run 'docs-builder format --write' to automatically fix all instances.
mapped_pages:
- https://www.elastic.co/guide/en/logstash/current/ea-integrations.html

navigation_title: Using Logstash with Elastic integrations
applies_to:
stack: ga
serverless: ga
products:
- id: elasticsearch
- id: cloud-hosted
- id: cloud-enterprise
- id: cloud-kubernetes
---

# Using Logstash with Elastic integrations [ea-integrations]
Expand Down Expand Up @@ -30,6 +40,7 @@

::::{admonition} How to
Create a {{ls}} pipeline that uses the [elastic_agent input](logstash-docs-md://lsr/plugins-inputs-elastic_agent.md) plugin, and the [elastic_integration filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) plugin as the *first* filter in your {{ls}} pipeline. You can add more filters for additional processing, but they must come after the `logstash-filter-elastic_integration` plugin in your configuration. Add one or more output plugins to complete your pipeline.

::::


Expand Down
Loading