[FIXED] microservice cleanup (flapping MicroServiceStops... tests) by levb · Pull Request #816 · nats-io/nats.c

levb · 2024-11-01T16:48:16Z

Removed the global hash of services, it was the most immediate source of the deadlock.
Added the list of microservices to natsConnection. NOTE increased memory allocation for all connections, need a better way.
Fixed the flow of replacing an existing endpoint in a service.
Got rid of ep->name since it was synonimous with ep->cfg->Name
Adjusted the tests, some minor fixes.

- Removed the global hash of services, it was the most immediate source of the deadlock. - Added the list of microservices to `natsConnection`. **NOTE** increased memory allocation for all connections, need a better way. - Fixed the flow of replacing an existing endpoint in a service. - Got rid of ep->name since it was synonimous with ep->cfg->Name - Adjusted the tests, some minor fixes.

…-when-server-stops

codecov · 2024-11-01T16:52:32Z

Codecov Report

Attention: Patch coverage is 53.33333% with 98 lines in your changes missing coverage. Please review.

Project coverage is 70.50%. Comparing base (1553d4a) to head (a15d5a8).
Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
src/micro.c	54.54%	41 Missing and 49 partials ⚠️
src/micro_endpoint.c	14.28%	3 Missing and 3 partials ⚠️
src/conn.c	75.00%	0 Missing and 1 partial ⚠️
src/micro_monitoring.c	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #816      +/-   ##
==========================================
+ Coverage   68.71%   70.50%   +1.79%     
==========================================
  Files          39       47       +8     
  Lines       15207    15366     +159     
  Branches     3143     3177      +34     
==========================================
+ Hits        10449    10834     +385     
+ Misses       1700     1460     -240     
- Partials     3058     3072      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kozlovic · 2024-11-04T16:26:24Z

    // Wrap the connection callbacks before we subscribe to anything.
-    MICRO_CALL(err, _wrap_connection_event_callbacks(m));
+    MICRO_CALL(err, micro_ErrorFromStatus(
+                        natsOptions_setMicroCallbacks(m->nc->opts, _on_connection_closed, _on_error)));


So for instance even if the nc connection is valid at the time we enter micro_AddService, but happen to be closed during execution of this function, it will leave things in a bad state. If service is part of a connection, I think it should be handled by the connection, even if micro_X calls are just a proxy to internal natsConn_X calls. Have you thought that way?

I did, in fact I was VERY inclined to make them "1st class citizens" in natsConnection, but not for this PR... This all really came out from trying to fix a single race condition on that global services table I had before, wrong locking order in 1 place.

OT: In light of the orbit movement I've been contemplating adding simple "internal use" extensibility to both Connection and Subscription; for data (like a hashmap) and callbacks (say, with a bool lock parameter). My initial microservices implementation was naive, it is still marked experimental, and I would really love to externalize and re-factor it to orbit before it is final.

I moved all "attach to connection" functionality into the func, and added a natsConn_Lock around + connection validation. Thanks for pointing it out. After the service is added to connection, it should unwind "normally" if it's terminated.

kozlovic · 2024-11-04T16:38:22Z

+    numEndpoints = m->numEndpoints;
+    _unlock_service(m);
+
+    if ((refs == 0) && (numEndpoints == 0))


Question: would it be symptomatic of a bug if we have refs == 0 and numEndpoints != 0?

No, because of the order of execution. As I understand it (and can prove, I think) On(Connection)Closed may be invoked in the midst of Sub's threads calling the On(Sub)Complete from their own threads. greyhair ++

Similar for error callbacks, maybe it was just one of the 2, not 100% sure now.

kozlovic · 2024-11-05T17:08:24Z

+        nc->services[nc->numServices] = service;
+        nc->numServices++;
    }
+    natsMutex_Unlock(nc->servicesMu);


The lock was acquired only under if (s == NATS_OK) after setting the callbacks, but here you unlock unconditionally. You should the if statement above (where you bump service's ref count) under the if statement where you acquire the service lock. This unlock would fall under that previous if statement too.

(hit the wrong button before) thanks for catching this!

kozlovic

LGTM

…g MicroServiceStops... tests) (#816)

…ats-io#816) * [FIXED] microservice cleanup (flapping MicroServiceStops... tests) - Removed the global hash of services, it was the most immediate source of the deadlock. - Added the list of microservices to `natsConnection`. **NOTE** increased memory allocation for all connections, need a better way. - Fixed the flow of replacing an existing endpoint in a service. - Got rid of ep->name since it was synonimous with ep->cfg->Name - Adjusted the tests, some minor fixes. * Adjusted test parameters, REPEAT micro on travis * fixes, cleannup * Restored Travis * PR Feedback: include EP name in the error description * PR Feedback: s/not_used/ignored * PR Feedback: better connection handling in AddService * PR Feedback: removed extra indent * PR Feedback: just move the last array element, do not copy * PR feedback: removed unneeded return * PR feedback: quote display names in errors (micro) * PR feedback: removed trailing whitespace in micro* * PR feedback

…g MicroServiceStops... tests) (nats-io#816)

levb · 2024-12-09T17:13:36Z

Cherry-picked into v3.9.2

Lev Brouk added 2 commits November 1, 2024 09:38

Merge branch 'main' of github.com:nats-io/nats.c into lev-micro-stops…

a15995b

…-when-server-stops

levb commented Nov 1, 2024

View reviewed changes

Comment thread src/micro_endpoint.c

levb commented Nov 1, 2024

View reviewed changes

Comment thread src/micro_endpoint.c

levb commented Nov 1, 2024

View reviewed changes

Comment thread src/natsp.h

levb commented Nov 1, 2024

View reviewed changes

Comment thread test/test.c

Adjusted test parameters, REPEAT micro on travis

bc40ea7

levb changed the title ~~[FIXED] microservice cleanup (flapping MicroServiceStops... tests)~~ WIP: [FIXED] microservice cleanup (flapping MicroServiceStops... tests) Nov 1, 2024

Lev Brouk added 2 commits November 3, 2024 16:25

fixes, cleannup

8479c08

Restored Travis

026dde1

levb changed the title ~~WIP: [FIXED] microservice cleanup (flapping MicroServiceStops... tests)~~ [FIXED] microservice cleanup (flapping MicroServiceStops... tests) Nov 4, 2024

levb requested a review from kozlovic November 4, 2024 00:46

levb marked this pull request as ready for review November 4, 2024 00:46

kozlovic reviewed Nov 4, 2024

View reviewed changes

Lev Brouk added 8 commits November 5, 2024 07:16

PR Feedback: include EP name in the error description

89b05fb

PR Feedback: s/not_used/ignored

8c3b921

PR Feedback: better connection handling in AddService

53d5cb7

PR Feedback: removed extra indent

df42953

PR Feedback: just move the last array element, do not copy

0d96050

PR feedback: removed unneeded return

5fd559b

PR feedback: quote display names in errors (micro)

0a197ab

PR feedback: removed trailing whitespace in micro*

9bc5e7f

levb requested a review from kozlovic November 5, 2024 15:45

kozlovic requested changes Nov 5, 2024

View reviewed changes

PR feedback

a15d5a8

kozlovic approved these changes Nov 5, 2024

View reviewed changes

levb added the cherry-pick label Nov 5, 2024

levb merged commit 6c7b2cc into main Nov 5, 2024

levb deleted the lev-micro-stops-when-server-stops branch November 5, 2024 18:02

github-actions Bot pushed a commit that referenced this pull request Nov 5, 2024

[skip ci] Update docs: 6c7b2cc: [FIXED] microservice cleanup (flappin…

5d4900d

…g MicroServiceStops... tests) (#816)

github-actions Bot pushed a commit to levb/nats.c that referenced this pull request Dec 9, 2024

[skip ci] Update docs: 21acc14: [FIXED] microservice cleanup (flappin…

342450d

…g MicroServiceStops... tests) (nats-io#816)

levb removed the cherry-pick label Dec 9, 2024

levb mentioned this pull request Feb 28, 2025

Release v3.10 #853

Merged

Uh oh!

Conversation

levb commented Nov 1, 2024

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kozlovic Nov 4, 2024

Choose a reason for hiding this comment

Uh oh!

levb Nov 4, 2024

Choose a reason for hiding this comment

Uh oh!

levb Nov 5, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kozlovic Nov 4, 2024

Choose a reason for hiding this comment

Uh oh!

levb Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kozlovic Nov 5, 2024

Choose a reason for hiding this comment

Uh oh!

levb Nov 5, 2024

Choose a reason for hiding this comment

Uh oh!

kozlovic left a comment

Choose a reason for hiding this comment

Uh oh!

levb commented Dec 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Nov 1, 2024 •

edited

Loading

levb Nov 4, 2024 •

edited

Loading