Skip to content

Conversation

@gaobinlong
Copy link
Contributor

@gaobinlong gaobinlong commented May 20, 2020

Relates to #55682 and #51046.
The main point of this PR is adding a copy ingest processor which can copy the value of one field to another and even the top level of the document. The supported data types which can be copied are boolean, number, array, object, string, date.

@astefan astefan added the :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP label May 20, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label May 20, 2020
@danhermann danhermann self-requested a review May 20, 2020 11:08
@danhermann
Copy link
Contributor

@gaobinlong, thank you for this contribution. It's useful functionality that we would like to incorporate into ingest pipelines. We are reconsidering how we've organized the processors and are trying to determine whether this fits best as part of the existing set processor or as its own separate processor. If you'll bear with us while we consider that, we will let you know and then work to get this PR refactored if necessary and then merged.

@gaobinlong
Copy link
Contributor Author

@danhermann, I have tried to use set processor to copy one field to another, but found that we cannot copy array,object and boolean, because the template snippet always produces string value. I'm eager to know your idea about that and glad to do more work.

@danhermann
Copy link
Contributor

@gaobinlong, apologies for the delay on this one. We decided that this is how the set processor should work instead of producing string values as you note that it does now. If you would like to continue this PR by modifying the set processor to correctly handle, I would be happy to review it.

@gaobinlong
Copy link
Contributor Author

@danhermann, I'm glad to continue doing that and will refractor the code soon.

@gaobinlong
Copy link
Contributor Author

@danhermann , I have refactored the code and opened a new PR #63540, this PR can be closed lately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants