Skip to content

New log collector module #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jefftlin merged 61 commits into from
Dec 15, 2022
Merged

New log collector module #54

jefftlin merged 61 commits into from
Dec 15, 2022

Conversation

@Davidhua1996
Copy link
Collaborator

@Davidhua1996 Davidhua1996 commented Dec 14, 2022

What is the purpose of the change

Embedded log collector plugin for Flink and Spark Streaming application

Brief change log

  • Define a new module for collecting log of Streamis
  • Enhance the restful api of job to receive heartbeat request from Spark Streaming.
  • Define a log server to listen and store the log event.

Verifying this change

This change added tests and can be verified as follows:
run the test unit StreamisLogAppenderTest.java

Does this pull request potentially affect one of the following parts:

  • Dependencies: no
  • Anything that affects deployment: no
  • The Core framework, i.e., JobManager, Server.: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? not documented

Davidhua1996 and others added 30 commits October 17, 2022 10:57
@Davidhua1996
Support the method to get engine conn resource information.
c5ab969
@Davidhua1996
Init the structure of log module.
9f6b0be
@wushengyeyouya
Add the new feature for adding task and stop task for existed jobs.
a04b58d
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
0576382 
…v-0.2.3-log-collector
@Davidhua1996
Use the "\u0001" instead of "\0x001" to act as blank placeholder.
3afd19c
@Davidhua1996
SendBuffer and BucketConfig.
9288a1c
@Davidhua1996
Merge branch 'dev-0.2.3-log-collector' of github.com:WeDataSphere/Str…
8a753d3 
…eamis into dev-0.2.3-log-collector
@wushengyeyouya
Add the new feature for adding task and stop task for existed jobs.
a7183e9
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
1efbf47 
…v-0.2.3-log-collector
@Davidhua1996
Complete the log collector (80%).
ba4189d
@Davidhua1996
Merge branch 'dev-0.2.3-log-collector' of github.com:WeDataSphere/Str…
25e4646 
…eamis into dev-0.2.3-log-collector
@wushengyeyouya
fix the bug for adding task for existed jobs.
b1cf642
@Davidhua1996
Merge branch 'dev-0.2.3-log-collector' of github.com:WeDataSphere/Str…
e704f63 
…eamis into dev-0.2.3-log-collector
@wushengyeyouya
1. Add the ability for updating task for existed jobs.
247d93b 
2. Spark Streaming Job will not ask Linkis for status.
@Davidhua1996
RPC(Http) module classes in collector
f001612
@Davidhua1996
Merge branch 'dev-0.2.3-log-collector' of github.com:WeDataSphere/Str…
d4bd7c8 
…eamis into dev-0.2.3-log-collector
@wushengyeyouya
1. Add the ability for updating task for existed jobs.
06a4f63 
2. Spark Streaming Job will not ask Linkis for status.
@wushengyeyouya
1. Add the ability for updating task for existed jobs.
f5dc4ec 
2. Spark Streaming Job will not ask Linkis for status.
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
321bf48 
…v-0.2.3-log-collector
@Davidhua1996
The strategy to retry the request and compact the buffer in log colle…
f4e3cf5 
…ctor.
@Davidhua1996
Merge branch 'dev-0.2.3-log-collector' of github.com:WeDataSphere/Str…
aef0b89 
…eamis into dev-0.2.3-log-collector
@Davidhua1996
Complete the log collector (95%)
6153eb9
@Davidhua1996
Fix the problem in send buffer and consumer.
7ac4336
@wushengyeyouya
Optimize the architecture of fetch Job details.
9588e90
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
467d06f 
…v-0.2.3-log-collector
@Davidhua1996
Add config autowired interface to set the log appender params.
f3bee02
@Davidhua1996
Merge branch 'dev-0.2.3-log-collector' of github.com:WeDataSphere/Str…
58fe439 
…eamis into dev-0.2.3-log-collector
@wushengyeyouya
fix the compilation bug.
b31a319
@wushengyeyouya
fix the long to int parse bug.
6ac3092
@Davidhua1996
Complete the log collector (100%)
58d8410
Davidhua1996 and others added 24 commits October 24, 2022 01:26
@Davidhua1996
Fix serialization and configuration problem.
e0d9804
@wushengyeyouya
Optimize the update task and stop task logic.
b2052e1
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
d49ebc4 
…v-0.2.3-log-collector
@Davidhua1996
Fix the problem in fetching user info when the flink application does…
be6ef36 
…n't have hadoop dependencies.
@Davidhua1996
Merge branch 'dev-0.2.3-log-collector' of github.com:WeDataSphere/Str…
3359556 
…eamis into dev-0.2.3-log-collector
@Davidhua1996
Add new strategy named ThresholdFilter
7f449ee
@Davidhua1996
Fix the problem when closing bucket.
8b7903e
@Davidhua1996
Split the log collector into log4j1 and log4j2; Avoid the jackson mod…
cc67dbb 
…ule in log collector.
@Davidhua1996
Add module to collect spark container log.
f8dbed0
@wushengyeyouya
Add the new requestBody parameter projectName for adding task and s…
8d006e1 
…top task for existed jobs.
@wushengyeyouya
Add the new requestBody parameter projectName for adding task and s…
c518e10 
…top task for existed jobs.
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
f40985c 
…v-0.2.3-log-collector
@Davidhua1996
Complete the strategy in JsonTool.escapeStrValue().
49c1f95
@wushengyeyouya
resolve 413 response status for update task.
d882cf4
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
86d8ad6 
…v-0.2.3-log-collector
@wushengyeyouya
resolve 413 response status for update task.
d33b0f3
@Davidhua1996
Fix the in method restoreInvariants of AbstractRpcLogSender.
11cf7c7
@Davidhua1996
Merge branch 'dev-0.2.3-log-collector' of github.com:WeDataSphere/Str…
4e330a2 
…eamis into dev-0.2.3-log-collector
@wushengyeyouya
resolve 413 response status for update task.
e9c8792
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
2004b57 
…v-0.2.3-log-collector
@Davidhua1996
adjust the relation path in pom.xml.
b3d60fe
@Davidhua1996
Fix the problem in pom.xml of stream job log modules.
67defce
@wushengyeyouya
resolve the LinkisJobInfo is null for adding task.
02dcdc6
@wushengyeyouya
Merge remote-tracking branch 'origin/dev-0.2.3-log-collector' into de…
a4fc531 
…v-0.2.3-log-collector
@Davidhua1996
Copy link
Collaborator Author

Davidhua1996 commented Dec 14, 2022

related issue #53

@jefftlin
Copy link
Collaborator

jefftlin commented Dec 15, 2022

LGTM！

@jefftlin jefftlin merged commit 21cd68b into WeBankFinTech:dev-0.2.3 Dec 15, 2022
Alexkun pushed a commit to WeDataSphere/Streamis that referenced this pull request Jun 5, 2024
@jefftlin
Merge pull request WeBankFinTech#54 from WeDataSphere/dev-0.2.3-log-c…
1443aee 
…ollector

New log collector module
jiaboy20 pushed a commit to jiaboy20/Streamis that referenced this pull request Mar 4, 2025
@jefftlin
Merge pull request WeBankFinTech#54 from WeDataSphere/dev-0.2.3-log-c…
e0a7dd1 
…ollector

New log collector module
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Davidhua1996 @jefftlin @wushengyeyouya