Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky capture service start #20024

Merged
merged 4 commits into from
Aug 29, 2024
Merged

Fix flaky capture service start #20024

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dc2e7bc
Initial commit
zbud-msft Aug 23, 2024
3d5f62d
Remove failure from initial do_capture read
zbud-msft Aug 27, 2024
967caa8
Drop messages until proxy is setup
zbud-msft Aug 27, 2024
afa2e6b
Remove control character tracking from do_capture
zbud-msft Aug 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions src/sonic-eventd/src/eventd.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,7 @@ capture_service::do_capture()
zmq_msg_t msg;
zmq_msg_init(&msg);
int rc = zmq_msg_recv(&msg, cap_sub_sock, 0);
RET_ON_ERR(rc == 1, "Failed to read subscription message when XSUB connects to XPUB");

/*
* When XSUB socket connects to XPUB, a subscription message is sent as a single byte 1.
* When capture service begins to read, the very first message that it will read is this
Expand All @@ -409,8 +409,30 @@ capture_service::do_capture()
*
* This behavior will only happen once when XSUB connects to XPUB not everytime cache is started.
*
* There are chances that there are events already published to XSUB endpoint before XSUB is able to connect to XPUB, so we can receive events
before the subscription message
*/
init_done = true;


if(rc == 1) { // Expected case to receive subscription message as very first message
SWSS_LOG_INFO("Received subscription message when XSUB connects to XPUB");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are too much "received subscription message" log, suggest change to SWSS_LOG_DEBUG.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will happen only once during process lifetime.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#Closed

} else if (rc > 1) { // If there are events already published to XSUB before XSUB connects to XPUB, we can receive events before subscription message
string event_source((const char*)zmq_msg_data(&msg), zmq_msg_size(&msg));
SWSS_LOG_DEBUG("Receiving event from source: %s, will read second part of event", event_source.c_str());
int more = 0;
size_t more_size = sizeof(more);
zmq_getsockopt(cap_sub_sock, ZMQ_RCVMORE, &more, &more_size);
if(more) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a loop here to read all message?

Also, if there is an ADO please update the PR description.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed logic to not account for capture service capturing control character. Will be done in zmq_read_part

sonic-net/sonic-swss-common#906

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#Closed

zmq_msg_t msg_part;
zmq_msg_init(&msg_part);
zmq_msg_recv(&msg_part, cap_sub_sock, 0);
zmq_msg_close(&msg_part);
}
} else {
SWSS_LOG_ERROR("Error reading from ZMQ socket, rc=%d", rc);
}
zmq_msg_close(&msg);
init_done = true;
}

while (m_ctrl != START_CAPTURE) {
Expand Down
23 changes: 6 additions & 17 deletions src/sonic-eventd/tests/eventd_ut.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,9 +150,8 @@ static const test_data_t ldata[] = {
},
};


void run_cap(void *zctx, bool &term, string &read_source,
int &cnt, bool &should_read_control)
int &cnt)
{
void *mock_cap = zmq_socket (zctx, ZMQ_SUB);
string source;
Expand All @@ -165,11 +164,10 @@ void run_cap(void *zctx, bool &term, string &read_source,
EXPECT_EQ(0, zmq_setsockopt(mock_cap, ZMQ_SUBSCRIBE, "", 0));
EXPECT_EQ(0, zmq_setsockopt(mock_cap, ZMQ_RCVTIMEO, &block_ms, sizeof (block_ms)));

if(should_read_control) {
zmq_msg_t msg;
zmq_msg_init(&msg);
EXPECT_NE(1, zmq_msg_recv(&msg, mock_cap, 0)); // Subscription message should be read by do_capture
}
zmq_msg_t msg;
zmq_msg_init(&msg);
int rc = zmq_msg_recv(&msg, mock_cap, 0);
EXPECT_EQ(1, rc); // read control character

while(!term) {
string source;
Expand Down Expand Up @@ -228,7 +226,6 @@ void run_pub(void *mock_pub, const string wr_source, internal_events_lst_t &lst)
TEST(eventd, proxy)
{
printf("Proxy TEST started\n");
bool should_read_control = false;
bool term_sub = false;
bool term_cap = false;
string rd_csource, rd_source, wr_source("hello");
Expand All @@ -246,7 +243,7 @@ TEST(eventd, proxy)
EXPECT_EQ(0, pxy->init());

/* capture in a thread */
thread thrc(&run_cap, zctx, ref(term_cap), ref(rd_csource), ref(rd_cevts_sz), ref(should_read_control));
thread thrc(&run_cap, zctx, ref(term_cap), ref(rd_csource), ref(rd_cevts_sz));

/* subscriber in a thread */
thread thr(&run_sub, zctx, ref(term_sub), ref(rd_source), ref(rd_evts), ref(rd_evts_sz));
Expand Down Expand Up @@ -283,17 +280,9 @@ TEST(eventd, proxy)

zmq_close(mock_pub);

/* Do control test */

should_read_control = true;

/* capture in a thread */
thread thrcc(&run_cap, zctx, ref(term_cap), ref(rd_csource), ref(rd_cevts_sz), ref(should_read_control));

delete pxy;
pxy = NULL;

thrcc.join();
zmq_ctx_term(zctx);

/* Provide time for async proxy removal to complete */
Expand Down
Loading