Remove sleep and change expectations for test_submit_exceed_max_rate

[roryharr]

Problem

CI is failing intermittently during this test since #7196.
The test is dispatching 120 points, waiting 2 seconds, expecting to see 100 of those points written as it is capped by the max write rate, and then a final point written when the flush is called at the end of the test.

Before the change the receiver thread woke up whenever new data was received or a timeout occurred at write_frequency/2 (write_frequency is 1s in this test), while now the receiver thread wakes up every 5ms regardless of whether data is present or not. In both cases, whenever the receiver thread wakes up, it checks to see if the duration since the last write time is greater than write_frequency and logs data if it is.

Prior to the change, with the 2 second wait the thread would not log another data point prior to flush. I printed the log times and could see the wakeup prior to the flush occurs at duration since last write: 505.140834ms, meaning it is likely always taking slightly longer than 1s. This leads to two full intervals not fitting into 1s.

With the change and the 5ms wakeup, the thread does wakeup a second time, leading to an extra metrics write, causing the test to fail

Summary of Changes

• Reduced wait between starting metrics gathering and flushing from 2s to 1500ms.

Fixes #

[brooksprumo]

to unblock master.

[steviez]

I'm looking at this one right now, give me a bit to finish reviewing

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.8%. Comparing base (1286a6a) to head (a12ad77).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7313   +/-   ##
=======================================
  Coverage    82.8%    82.8%           
=======================================
  Files         801      801           
  Lines      363318   363315    -3     
=======================================
+ Hits       300877   300886    +9     
+ Misses      62441    62429   -12     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[steviez]

Rory and I jumped on a call to discuss this one real quick. Reducing the wait addressed the issue, but still left this test racy.

Instead, we removed the sleep and took advantage of the Barrier that MetricsAgent::flush() creates. The result is no more raciness + no more sleep.

[steviez]

@roryharr - Since we changed direction with the PR, can you change the title ?