Synthetic source does not still support copy_to#652
Synthetic source does not still support copy_to#652salvatore-campagna merged 13 commits intoelastic:masterfrom
copy_to#652Conversation
|
Since |
When using logsdb index mode copy_to is disabled because it is not supported by synthetic source. So we need to change queries so to reflect the fact that the message field would be empty.
There was a problem hiding this comment.
It looks like our template engine does not process files in the workflows directory. As a result we are not able to conditionally execute the query on message or kubernetes.even.message. I will just unconditionally execute it using kubernetes.even.message that is the original field name. After we introduce support for copy_to we can restore the original behaviour.
This is done since copy_to does not work in synthetic source and the message field would be empty.
We also introduce a track parameter which we can use to select the workflow directory to use to read workflow queries. The default is `workflows` that is the existing one. Using `workflows-logsdb` allows us to use workflow queries which do not realy on the usage of copy_to used by the metricbeat template. This workaround is introduced to deal with synthetic source (and logsdb) not supporting copy_to. Once support for copy_to is available we will revert this change and rely on the existing and default workflows.
|
I tried adding a @gareth-ellis @charlie-pichette does it sound reasonable? We will revert this once synthetic source supports |
|
BTW this would have been much easier if this track used the standard |
martijnvg
left a comment
martijnvg
left a comment
There was a problem hiding this comment.
I left a few questions. This looks good to me. I think it is ok to temporarily copy the work flow files and undo this ounce we have copy_to support.
There was a problem hiding this comment.
Maybe just refer to the other README and indicate this is just a temp solutions for logsdb?
No need to repeat READMEs, right?
There was a problem hiding this comment.
And maybe do this for the other READMEs too?
There was a problem hiding this comment.
Shouldn't this be determined by the work_flow track param? Or is the idea that the work_flow param is determined by the index mode track param?
There was a problem hiding this comment.
the workflow parameter just says hosts, network or overview - the workflows-folder is what refers to either currently workflows, but for logsdb we want to use the workflows-logsdb folder
gareth-ellis
force-pushed
the
fix/security-message-copy-to
branch
from
d88ff33 to
ab2e5a4
Compare
|
I executed a test for the stateful case and double checkd all datastream backing indices are actually using LogsDB. |
|
The track completed succesfully |
I have no information about |
|
@salvatore-campagna Regarding the |
|
@elasticmachine update branch |
This reverts commit 2b42a38.
…astic#652)"" This reverts commit b8b2f27.
6 participants
This causes the
elastic/securitytrack to fail execution whenindex_modeis set tologsdb. This is happening because LogsDB uses synthetic source which, in turn, doesnot support
copy_to. Supportingcopy_tois expected to come in Elasticsearch 8.16.In the meanwhile we just exclude the
copy_tosetting from the mapping so to avoidtriggering the error.
This needs backporting to 8.15.