3. September 2024

[mkow]

Agenda

• Discuss the new design of Dump and Reset stats data on demand using SIGUSR1 signal #1857

Discuss the new design of #1857

Quick summary: (written by @dimakuv)

• Dmitrii explained the current design
• Michał found a new data race:
  • If the previous round of the protocol was terminated due to some thread exiting (and the epoch was incremented), and then immediately a new round of the protocol is initiated by user's SIGUSR1, then there are some "struggler" follower threads that still may receive/are working on the previous round and they will also receive a new SIGUSR1 from the new run (in this case, the follower will incorrectly increment followers_visited_num twice, even though the previous-round should have zeroed this variable out).
  • Dmitrii and Michał agree that for complete correctness, more metadata needs to be kept, i.e., we can't increment this fully correctly with only "summarized" integer variables. We need a map of tid -> <terminated|havent-yet-processed-signal|already-processed-signal>, which is very complex and also bad in performance.
• Don suggested to dump the asynchronous (distributed) design with leader broadcasting SIGUSR1 to followers:
  • Instead, the leader grabs a global spinlock (that prevents other threads to terminate) and iterates over the list of threads, finds their thread-local storage, and reads their SGX stats.
  • This may have worse accuracy, but we don't care about being very accurate.
• Dmitrii: do we even need to print thread-local statistics? We should probably drop printing them altogether (but keep thread-local stats for performance reasons, since atomically updating global counters is too expensive, especially in case of NUMA nodes).