Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SPL processor in ilogtail for enhanced data processing #1278

Merged
merged 11 commits into from
Dec 26, 2023

Conversation

linrunqi08
Copy link
Collaborator

@linrunqi08 linrunqi08 commented Dec 25, 2023

需要重点关注的文件:

core/reader/SourceBuffer.h 这里把原内存池实现替换了一下,主要参考的是spl的内存池实现
core/aggregator/Aggregator.cpp 这里把原来Add函数修改了一下,SPL会使用LOGSOTE模式进行聚合,聚合就是使用list的形式进行发送。
core/processor/daemon/LogProcess.cpp 这里PipelineEventGroup转logGroup的时候,之前接口参数改成list的时候有遗漏,现在补充上了

This commit introduces support for the SLS Processing Language (SPL) within the ilogtail observability collector, allowing users to leverage SPL for data processing and explorative querying. SPL, a proprietary data processing language developed by Alibaba Cloud for its Log Service (SLS), adopts a shell-like pipeline symbol (|) approach that facilitates sequential data processing and analysis.

With the integration of the SPL Processor, ilogtail users can now apply SPL's powerful data transformation and querying capabilities directly within the observability collector. This feature enhances the flexibility and efficiency of data handling in observability scenarios, offering a familiar and intuitive syntax for users already accustomed to shell operations.

However, it is important to note that the current implementation imposes a limitation where the SPL Processor cannot be configured in conjunction with other C++ processors. This restriction is in place to ensure compatibility and stability of the processing pipeline within ilogtail.

For additional information on SPL and its functionalities, please refer to the official documentation: https://help.aliyun.com/zh/sls/user-guide/spl-overview.

This SPL support marks a significant step forward in expanding ilogtail’s processing capabilities and in providing users with more versatile tools for data analysis within the Alibaba Cloud ecosystem.

@linrunqi08 linrunqi08 changed the title Feature/taiye/spl merge Support SPL Processor Dec 25, 2023
.devcontainer/devcontainer.json Outdated Show resolved Hide resolved
.devcontainer/Dockerfile Outdated Show resolved Hide resolved
.github/workflows/build-core-ut.yaml Show resolved Hide resolved
core/dependencies.cmake Outdated Show resolved Hide resolved
core/dependencies.cmake Outdated Show resolved Hide resolved
core/processor/ProcessorSPL.cpp Show resolved Hide resolved
core/processor/ProcessorSPL.cpp Outdated Show resolved Hide resolved
core/reader/SourceBuffer.h Outdated Show resolved Hide resolved
core/spl/PipelineEventGroupOutput.cpp Outdated Show resolved Hide resolved
core/spl/PipelineEventGroupOutput.cpp Outdated Show resolved Hide resolved
core/dependencies.cmake Outdated Show resolved Hide resolved
@yyuuttaaoo yyuuttaaoo added the feature New feature label Dec 26, 2023
@yyuuttaaoo yyuuttaaoo added this to the v2.0 milestone Dec 26, 2023
@linrunqi08 linrunqi08 merged commit 6c41454 into main Dec 26, 2023
21 checks passed
@yyuuttaaoo yyuuttaaoo changed the title Support SPL Processor Support SPL processor in ilogtail for enhanced data processing Dec 26, 2023
@nelsontang
Copy link

编译错误:fatal error: spl/rw/IO.h: No such file or directory
请问是否有依赖库地址?

@linrunqi08 linrunqi08 deleted the feature/taiye/spl_merge branch May 20, 2024 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants