-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support SPL processor in ilogtail for enhanced data processing #1278
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
linrunqi08
commented
Dec 25, 2023
yyuuttaaoo
reviewed
Dec 25, 2023
yyuuttaaoo
reviewed
Dec 26, 2023
yyuuttaaoo
approved these changes
Dec 26, 2023
yyuuttaaoo
changed the title
Support SPL Processor
Support SPL processor in ilogtail for enhanced data processing
Dec 26, 2023
编译错误:fatal error: spl/rw/IO.h: No such file or directory |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
需要重点关注的文件:
core/reader/SourceBuffer.h 这里把原内存池实现替换了一下,主要参考的是spl的内存池实现
core/aggregator/Aggregator.cpp 这里把原来Add函数修改了一下,SPL会使用LOGSOTE模式进行聚合,聚合就是使用list的形式进行发送。
core/processor/daemon/LogProcess.cpp 这里PipelineEventGroup转logGroup的时候,之前接口参数改成list的时候有遗漏,现在补充上了
This commit introduces support for the SLS Processing Language (SPL) within the ilogtail observability collector, allowing users to leverage SPL for data processing and explorative querying. SPL, a proprietary data processing language developed by Alibaba Cloud for its Log Service (SLS), adopts a shell-like pipeline symbol (|) approach that facilitates sequential data processing and analysis.
With the integration of the SPL Processor, ilogtail users can now apply SPL's powerful data transformation and querying capabilities directly within the observability collector. This feature enhances the flexibility and efficiency of data handling in observability scenarios, offering a familiar and intuitive syntax for users already accustomed to shell operations.
However, it is important to note that the current implementation imposes a limitation where the SPL Processor cannot be configured in conjunction with other C++ processors. This restriction is in place to ensure compatibility and stability of the processing pipeline within ilogtail.
For additional information on SPL and its functionalities, please refer to the official documentation: https://help.aliyun.com/zh/sls/user-guide/spl-overview.
This SPL support marks a significant step forward in expanding ilogtail’s processing capabilities and in providing users with more versatile tools for data analysis within the Alibaba Cloud ecosystem.