[management/client] Integrate Job API with Job Stream and Client Engine by aliamerj · Pull Request #4428 · netbirdio/netbird

aliamerj · 2025-09-01T16:08:52Z

Describe your changes

This PR connects the Job API with the peer job stream and updates both server and client code to work together.

What was done:

Integrated Job API with the JobManager stream channel.
Updated server logic to send and track jobs through the channel.
Implemented client-side handling in the engine for job requests and responses.
Adjusted existing code to align with the new unified job flow.

Issue ticket number and link #4354

Stack

feature/remote-debug - ⚠️ No PR associated with branch
- [management/client] Integrate Job API with Job Stream and Client Engine #4428 👈

Checklist

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

I added/updated documentation for this change
Documentation is not needed for this change (explain why)

Docs PR URL (required if "docs added" is checked)

Paste the PR link from https://github.com/netbirdio/docs here:

https://github.com/netbirdio/docs/pull/__

integrate api with stream and implement some client side

aliamerj · 2025-09-01T16:32:24Z

 	// blocking until error
 	err = c.receiveUpdatesEvents(stream, serverPubKey, msgHandler)
-	if err != nil {
+	if err != nil && err != io.EOF{


I noticed that when sending a job, I was repeatedly seeing logs from the sync with EFO. To fix this, I updated this not ideal but for now,
the problem is when sending stream request for peer both sync and job triggered and one get valid request and the other get nothing , so we get this error and disconnect the peer , I think we need we to drop the io.EOF check in sync as i did for the job

pascal-fischer · 2025-09-04T14:30:32Z

+			case *mgmProto.JobRequest_Bundle:
+				uploadKey, err := e.handleBundle(params.Bundle)
+				if err != nil {
+					return &mgmProto.JobResponse{
+						ID:     msg.ID,
+						Status: mgmProto.JobStatus_failed,
+						Reason: []byte(err.Error()),
+					}
+				}
+				return &mgmProto.JobResponse{
+					ID:     msg.ID,
+					Status: mgmProto.JobStatus_succeeded,
+					WorkloadResults: &mgmProto.JobResponse_Bundle{
+						Bundle: &mgmProto.BundleResult{
+							UploadKey: uploadKey,
+						},


Here you are creating unnecessary code duplication. The only generic part is WorkloadResults. The rest is identical and can be build after the switch

The generic part are

Status could be succeeded or failed

Reason could be nil or error message

WorkloadResults could be nil or the result
so the identical part is just job id

pascal-fischer · 2025-09-04T14:37:34Z

+	var job types.Job
+	if err := job.ApplyResponse(jobResponse); err != nil {
+		return status.Errorf(status.Internal, err.Error())
+	}


I would move this conversion to the GRPC layer to not drag proto sturct to deep into the code

pascal-fischer · 2025-09-05T09:01:46Z

+func (e *Engine) getPeerClient() (*grpc.ClientConn, error) {
+	conn, err := grpc.NewClient(
+		strings.TrimPrefix(e.daemonAddress, "tcp://"),
+		grpc.WithTransportCredentials(insecure.NewCredentials()),
+	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to connect to daemon error: %v\n"+
+			"If the daemon is not running please run: "+
+			"\nnetbird service install \nnetbird service start\n", err)
+	}
+
+	return conn, nil
+}
+


I don't think this is the way to go. The engine is part of the daemon so we are basically connecting to ourself.

yes, i understand , that is why i needed help to understand what is the best why to call bundle ,
is passing grpc client and call it from it or passing address string and call the bundle or without the grpc and call the debugging directly ?
and also considering the other os

…4469)

…rview

aliamerj · 2025-09-15T13:17:46Z

 	// todo do not throw error in case of cancelled context
 	ctx = internal.CtxInitState(ctx)
-	c.connectClient = internal.NewConnectClient(ctx, cfg, c.recorder)
+	c.connectClient = internal.NewConnectClient(ctx, cfg, c.recorder, "")


Do we need to pass the logFile here

aliamerj · 2025-09-15T13:18:28Z


-	connectClient := internal.NewConnectClient(ctx, config, r)
+	//todo: do we need to pass logFile here ?
+	connectClient := internal.NewConnectClient(ctx, config, r, "")


Do we need to pass the logFile here

aliamerj · 2025-09-15T13:19:05Z

-	client := internal.NewConnectClient(ctx, c.config, recorder)
+
+	//todo: do we need to pass logFile here ?
+	client := internal.NewConnectClient(ctx, c.config, recorder, "")


Do we need to pass the logFile here

aliamerj · 2025-09-15T13:20:37Z


-	c.connectClient = internal.NewConnectClient(ctx, cfg, c.recorder)
+	//todo: do we need to pass logFile here ?
+	c.connectClient = internal.NewConnectClient(ctx, cfg, c.recorder, "")


Do we need to pass the logFile here

aliamerj · 2025-09-15T13:21:31Z

 	runOperation := func() error {
 		log.Tracef("running client connection")
-		s.connectClient = internal.NewConnectClient(ctx, config, statusRecorder)
+		s.connectClient = internal.NewConnectClient(ctx, config, statusRecorder, s.logFile)


here the only place we we pass the logFile

sonarqubecloud · 2025-09-18T09:42:12Z

Quality Gate passed

Issues
0 New issues
3 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

…ne (#4428) * integrate api integrate api with stream and implement some client side * fix typo and fix validation * use real daemon address * redo the connect via address * Refactor the debug bundle generator to be ready to use from engine (#4469) * fix tests * fix lint * fix bug with stream * try refactor status 1 * fix convert fullStatus to statusOutput & add logFile * fix tests * fix tests * fix not enough arguments in call to nbstatus.ConvertToStatusOutputOverview * fix status_test * fix(engine): avoid deadlock when stopping engine during debug bundle * use atomic for lock-free * use new lock --------- Co-authored-by: Zoltan Papp <zoltan.pmail@gmail.com>

integrate api

08641b7

integrate api with stream and implement some client side

aliamerj requested review from lixmal and pascal-fischer September 1, 2025 16:29

aliamerj commented Sep 1, 2025

View reviewed changes

fix typo and fix validation

171925d

aliamerj mentioned this pull request Sep 3, 2025

Add Frontend Support for Peer Debug Bundle Trigger and History netbirdio/dashboard#485

Merged

use real daemon address

e47e648

pascal-fischer reviewed Sep 5, 2025

View reviewed changes

redo the connect via address

dec307f

github-actions Bot mentioned this pull request Sep 9, 2025

Refactor the debug bundle generator to be ready to use from engine #4469

Merged

7 tasks

pappz and others added 4 commits September 9, 2025 18:12

Refactor the debug bundle generator to be ready to use from engine (#…

f1cc9e4

…4469)

fix tests

9cb9fc4

fix lint

7fba07b

fix bug with stream

7c60ad6

aliamerj requested a review from pascal-fischer September 10, 2025 12:25

aliamerj added 5 commits September 12, 2025 11:06

try refactor status 1

36215b8

fix convert fullStatus to statusOutput & add logFile

426a6f2

fix tests

3ee24d7

fix tests

b1077aa

fix not enough arguments in call to nbstatus.ConvertToStatusOutputOve…

d82d56a

…rview

aliamerj commented Sep 15, 2025

View reviewed changes

fix status_test

9d5694f

aliamerj requested a review from pappz September 16, 2025 14:49

aliamerj added 3 commits September 16, 2025 21:06

fix(engine): avoid deadlock when stopping engine during debug bundle

889c3c8

use atomic for lock-free

3027029

use new lock

2eba4af

pappz approved these changes Sep 18, 2025

View reviewed changes

pascal-fischer approved these changes Sep 23, 2025

View reviewed changes

aliamerj merged commit acec87d into feature/remote-debug Sep 23, 2025
37 of 38 checks passed

aliamerj deleted the remote-debug/integration branch September 23, 2025 11:07

Uh oh!

Conversation

aliamerj commented Sep 1, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes

What was done:

Issue ticket number and link #4354

Stack

Checklist

Documentation

Docs PR URL (required if "docs added" is checked)

Uh oh!

aliamerj Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aliamerj Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aliamerj Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aliamerj commented Sep 1, 2025 •

edited by github-actions Bot

Loading

aliamerj Sep 1, 2025 •

edited

Loading

aliamerj Sep 15, 2025 •

edited

Loading

aliamerj Sep 15, 2025 •

edited

Loading

sonarqubecloud Bot commented Sep 18, 2025 •

edited

Loading