Skip to content

Custom ingest pipelines #2094

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
bmorelli25 merged 6 commits into from
Aug 23, 2022
Merged

Custom ingest pipelines #2094

bmorelli25 merged 6 commits into from
Aug 23, 2022

Conversation

@bmorelli25
Copy link
Member

@bmorelli25 bmorelli25 commented Aug 17, 2022 •
edited
Loading

Summary

  • Moves the custom ILM tutorial to its own sub-page of the Data Streams topic
  • Adds a custom ingest pipeline tutorial that follows the same outline as the custom ILM tutorial

Screen Shot 2022-08-16 at 10 52 58 PM

Preview the new tutorial by clicking here.

Related

@bmorelli25 bmorelli25 added v8.4.0 backport-8.4 Automated backport with mergify labels Aug 17, 2022
@bmorelli25 bmorelli25 self-assigned this Aug 17, 2022
@ghost
Copy link

ghost commented Aug 17, 2022 •
edited by ghost
Loading

A documentation preview will be available soon:

bmorelli25
bmorelli25 commented Aug 17, 2022
View reviewed changes
docs/en/ingest-management/data-streams.asciidoc
Comment on lines +288 to +290
** Processor: "Set"
** Field: `test`
** Value: `true`
Copy link
Member Author

@bmorelli25 bmorelli25 Aug 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If anyone has a better idea for an example, I'm all ears.

Copy link
Member

@nchaulet nchaulet Aug 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally think it's an easy to understand example :)

@bmorelli25
Copy link
Member Author

bmorelli25 commented Aug 17, 2022 •
edited
Loading

@nchaulet / @joshdover here's a first draft of custom ingest pipeline documentation. I could use your help to:

  • make sure I'm on the right track here
  • ensure there aren't any important details missing
  • flesh out a better example if this one is too simple

@dedemorton you can hold off on a tech writer review for now

@bmorelli25 bmorelli25 marked this pull request as ready for review August 17, 2022 13:18
@bmorelli25 bmorelli25 requested a review from dedemorton as a code owner August 17, 2022 13:18
@bmorelli25 bmorelli25 requested review from dedemorton and removed request for dedemorton August 17, 2022 13:18
nchaulet
nchaulet reviewed Aug 18, 2022
View reviewed changes
@bmorelli25 bmorelli25 mentioned this pull request Aug 19, 2022
Merged
@bmorelli25
Copy link
Member Author

bmorelli25 commented Aug 23, 2022

I need to merge this now, but I welcome any additional feedback and will address it in a follow-up PR. Thanks!

@bmorelli25 bmorelli25 merged commit b5c3a1a into elastic:main Aug 23, 2022
@bmorelli25 bmorelli25 deleted the custom-ingest-pipelines branch August 23, 2022 15:36
mergify bot pushed a commit that referenced this pull request Aug 23, 2022
@bmorelli25 @mergify
Custom ingest pipelines (#2094)
1a97908 
* docs: incredibly _rough_ draft

* docs: clean 🧽🧽

* docs: remove notes

* docs: titles

* docs: fix build error

* docs: clarify what has a pipeline

(cherry picked from commit b5c3a1a)
@mergify mergify bot mentioned this pull request Aug 23, 2022
Merged
bmorelli25 added a commit that referenced this pull request Aug 23, 2022
@mergify @bmorelli25
Custom ingest pipelines (#2094) (#2122)
fe20d42 
* docs: incredibly _rough_ draft

* docs: clean 🧽🧽

* docs: remove notes

* docs: titles

* docs: fix build error

* docs: clarify what has a pipeline

(cherry picked from commit b5c3a1a)

Co-authored-by: Brandon Morelli <[email protected]>
@felixbarny
Copy link
Member

felixbarny commented Sep 8, 2022

I think there's a leftover of outdated docs that could be confusing to users: https://www.elastic.co/guide/en/elasticsearch/reference/8.4/ingest.html#pipeline-custom-logs-configuration

Maybe we should replace this with a link to https://www.elastic.co/guide/en/fleet/current/data-streams-pipeline-tutorial.html?

But maybe there are some subtleties here. Not sure if the latter would work for the "custom logs" integration which doesn't have a default pipeline.

cc @joshdover

@bmorelli25
Copy link
Member Author

bmorelli25 commented Sep 28, 2022

Catching up after a long vacation. Thanks for pointing this out, Felix! I'll connect with the ES docs team to figure out the best path forward and get this fixed.

AFAIK, the latter should work with all integrations. @joshdover can you verify?

@joshdover
Copy link
Contributor

joshdover commented Sep 28, 2022

@nchaulet can confirm, but my understanding is that it does work with custom logs

@nchaulet
Copy link
Member

nchaulet commented Sep 28, 2022

@nchaulet can confirm, but my understanding is that it does work with custom logs

Just checked and it does not work with custom logs as we do not generate a correct index template for that stream

bmorelli25 added a commit to bmorelli25/observability-docs that referenced this pull request Apr 11, 2023
@bmorelli25
Custom ingest pipelines (elastic#2094)
93a3953 
* docs: incredibly _rough_ draft

* docs: clean 🧽🧽

* docs: remove notes

* docs: titles

* docs: fix build error

* docs: clarify what has a pipeline
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nchaulet nchaulet nchaulet left review comments

@joshdover joshdover Awaiting requested review from joshdover

@dedemorton dedemorton Awaiting requested review from dedemorton

Assignees

@bmorelli25 bmorelli25

Labels

backport-8.4 Automated backport with mergify v8.4.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Request] Custom ingest pipeline in Fleet Document how to use custom ingest pipeline

4 participants

@bmorelli25 @felixbarny @joshdover @nchaulet