-
Notifications
You must be signed in to change notification settings - Fork 477
test(profiling): unflake test_gunicorn [backport 4.1]
#15864
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
|
0070691 to
8be17bc
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 250 ± 6 ms. The average import time from base is: 250 ± 3 ms. The import time difference between this PR and base is: 0.2 ± 0.2 ms. The difference is not statistically significant (z = 1.05). Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate backport-15860-to-4.1 (8be17bc) with baseline 4.1 (58e3115) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 18.159µs (SLO: <20.000µs -9.2%) vs baseline: 📈 +20.9% Memory: ✅ 42.802MB (SLO: <43.250MB 🟡 -1.0%) vs baseline: +5.2% ✅ add_inplace_aspectTime: ✅ 15.034µs (SLO: <20.000µs 📉 -24.8%) vs baseline: +0.2% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +3.9% ✅ add_inplace_noaspectTime: ✅ 0.338µs (SLO: <10.000µs 📉 -96.6%) vs baseline: +0.4% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.7% ✅ add_noaspectTime: ✅ 0.551µs (SLO: <10.000µs 📉 -94.5%) vs baseline: ~same Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ bytearray_aspectTime: ✅ 17.944µs (SLO: <30.000µs 📉 -40.2%) vs baseline: -0.3% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.7% ✅ bytearray_extend_aspectTime: ✅ 23.793µs (SLO: <30.000µs 📉 -20.7%) vs baseline: ~same Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ bytearray_extend_noaspectTime: ✅ 2.686µs (SLO: <10.000µs 📉 -73.1%) vs baseline: +1.4% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.5% ✅ bytearray_noaspectTime: ✅ 1.458µs (SLO: <10.000µs 📉 -85.4%) vs baseline: ~same Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.2% ✅ bytes_aspectTime: ✅ 16.741µs (SLO: <20.000µs 📉 -16.3%) vs baseline: ~same Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.1% ✅ bytes_noaspectTime: ✅ 1.406µs (SLO: <10.000µs 📉 -85.9%) vs baseline: -0.8% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.2% ✅ bytesio_aspectTime: ✅ 55.316µs (SLO: <70.000µs 📉 -21.0%) vs baseline: +0.6% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.7% ✅ bytesio_noaspectTime: ✅ 3.218µs (SLO: <10.000µs 📉 -67.8%) vs baseline: -0.2% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ capitalize_aspectTime: ✅ 14.728µs (SLO: <20.000µs 📉 -26.4%) vs baseline: +0.2% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +3.8% ✅ capitalize_noaspectTime: ✅ 2.564µs (SLO: <10.000µs 📉 -74.4%) vs baseline: +0.3% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ casefold_aspectTime: ✅ 14.720µs (SLO: <20.000µs 📉 -26.4%) vs baseline: +0.5% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.5% ✅ casefold_noaspectTime: ✅ 3.072µs (SLO: <10.000µs 📉 -69.3%) vs baseline: -2.5% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ decode_aspectTime: ✅ 15.632µs (SLO: <30.000µs 📉 -47.9%) vs baseline: +0.3% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.5% ✅ decode_noaspectTime: ✅ 1.599µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -0.4% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ encode_aspectTime: ✅ 18.395µs (SLO: <30.000µs 📉 -38.7%) vs baseline: 📈 +22.0% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.7% ✅ encode_noaspectTime: ✅ 1.467µs (SLO: <10.000µs 📉 -85.3%) vs baseline: -1.4% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.2% ✅ format_aspectTime: ✅ 171.036µs (SLO: <200.000µs 📉 -14.5%) vs baseline: ~same Memory: ✅ 43.037MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +5.1% ✅ format_map_aspectTime: ✅ 191.638µs (SLO: <200.000µs -4.2%) vs baseline: -0.2% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.1% ✅ format_map_noaspectTime: ✅ 3.747µs (SLO: <10.000µs 📉 -62.5%) vs baseline: -0.2% Memory: ✅ 42.644MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.9% ✅ format_noaspectTime: ✅ 3.186µs (SLO: <10.000µs 📉 -68.1%) vs baseline: +0.9% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.8% ✅ index_aspectTime: ✅ 15.316µs (SLO: <20.000µs 📉 -23.4%) vs baseline: +0.3% Memory: ✅ 42.841MB (SLO: <43.250MB 🟡 -0.9%) vs baseline: +5.1% ✅ index_noaspectTime: ✅ 0.462µs (SLO: <10.000µs 📉 -95.4%) vs baseline: +0.1% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.2% ✅ join_aspectTime: ✅ 16.993µs (SLO: <20.000µs 📉 -15.0%) vs baseline: +0.4% Memory: ✅ 42.900MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +5.6% ✅ join_noaspectTime: ✅ 1.475µs (SLO: <10.000µs 📉 -85.3%) vs baseline: -0.7% Memory: ✅ 42.664MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.6% ✅ ljust_aspectTime: ✅ 20.736µs (SLO: <30.000µs 📉 -30.9%) vs baseline: -0.7% Memory: ✅ 42.821MB (SLO: <43.250MB 🟡 -1.0%) vs baseline: +4.9% ✅ ljust_noaspectTime: ✅ 2.673µs (SLO: <10.000µs 📉 -73.3%) vs baseline: +0.9% Memory: ✅ 42.782MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.9% ✅ lower_aspectTime: ✅ 18.000µs (SLO: <30.000µs 📉 -40.0%) vs baseline: +0.1% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.3% ✅ lower_noaspectTime: ✅ 2.415µs (SLO: <10.000µs 📉 -75.9%) vs baseline: +2.1% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.6% ✅ lstrip_aspectTime: ✅ 17.692µs (SLO: <20.000µs 📉 -11.5%) vs baseline: ~same Memory: ✅ 42.664MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.2% ✅ lstrip_noaspectTime: ✅ 1.840µs (SLO: <10.000µs 📉 -81.6%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.4% ✅ modulo_aspectTime: ✅ 167.028µs (SLO: <200.000µs 📉 -16.5%) vs baseline: +0.3% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.0% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 174.952µs (SLO: <200.000µs 📉 -12.5%) vs baseline: +0.2% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.4% ✅ modulo_aspect_for_bytesTime: ✅ 168.369µs (SLO: <200.000µs 📉 -15.8%) vs baseline: ~same Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.2% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 171.649µs (SLO: <200.000µs 📉 -14.2%) vs baseline: +0.2% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.6% ✅ modulo_noaspectTime: ✅ 3.723µs (SLO: <10.000µs 📉 -62.8%) vs baseline: +0.3% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.4% ✅ replace_aspectTime: ✅ 214.755µs (SLO: <300.000µs 📉 -28.4%) vs baseline: ~same Memory: ✅ 42.920MB (SLO: <44.000MB -2.5%) vs baseline: +4.6% ✅ replace_noaspectTime: ✅ 5.156µs (SLO: <10.000µs 📉 -48.4%) vs baseline: -0.1% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.1% ✅ repr_aspectTime: ✅ 1.393µs (SLO: <10.000µs 📉 -86.1%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.4% ✅ repr_noaspectTime: ✅ 0.532µs (SLO: <10.000µs 📉 -94.7%) vs baseline: +1.1% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ rstrip_aspectTime: ✅ 23.021µs (SLO: <30.000µs 📉 -23.3%) vs baseline: 📈 +20.9% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.6% ✅ rstrip_noaspectTime: ✅ 1.889µs (SLO: <10.000µs 📉 -81.1%) vs baseline: -0.7% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ slice_aspectTime: ✅ 16.013µs (SLO: <20.000µs 📉 -19.9%) vs baseline: +0.2% Memory: ✅ 42.900MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +5.5% ✅ slice_noaspectTime: ✅ 0.601µs (SLO: <10.000µs 📉 -94.0%) vs baseline: +0.9% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.2% ✅ stringio_aspectTime: ✅ 53.829µs (SLO: <80.000µs 📉 -32.7%) vs baseline: ~same Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.5% ✅ stringio_noaspectTime: ✅ 3.574µs (SLO: <10.000µs 📉 -64.3%) vs baseline: +1.2% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ strip_aspectTime: ✅ 17.622µs (SLO: <20.000µs 📉 -11.9%) vs baseline: -0.6% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ strip_noaspectTime: ✅ 1.832µs (SLO: <10.000µs 📉 -81.7%) vs baseline: +0.1% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ swapcase_aspectTime: ✅ 18.460µs (SLO: <30.000µs 📉 -38.5%) vs baseline: +0.1% Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +5.2% ✅ swapcase_noaspectTime: ✅ 2.764µs (SLO: <10.000µs 📉 -72.4%) vs baseline: ~same Memory: ✅ 42.487MB (SLO: <43.500MB -2.3%) vs baseline: +4.6% ✅ title_aspectTime: ✅ 18.299µs (SLO: <30.000µs 📉 -39.0%) vs baseline: +0.3% Memory: ✅ 42.782MB (SLO: <43.000MB 🟡 -0.5%) vs baseline: +5.2% ✅ title_noaspectTime: ✅ 2.640µs (SLO: <10.000µs 📉 -73.6%) vs baseline: +0.8% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.6% ✅ translate_aspectTime: ✅ 20.538µs (SLO: <30.000µs 📉 -31.5%) vs baseline: +0.2% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.7% ✅ translate_noaspectTime: ✅ 4.239µs (SLO: <10.000µs 📉 -57.6%) vs baseline: -0.2% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ upper_aspectTime: ✅ 17.985µs (SLO: <30.000µs 📉 -40.1%) vs baseline: +0.2% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ upper_noaspectTime: ✅ 2.376µs (SLO: <10.000µs 📉 -76.2%) vs baseline: +0.6% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.0% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.159µs (SLO: <10.000µs 📉 -48.4%) vs baseline: 📈 +21.7% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% ✅ ospathbasename_noaspectTime: ✅ 4.308µs (SLO: <10.000µs 📉 -56.9%) vs baseline: +0.3% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ ospathjoin_aspectTime: ✅ 6.197µs (SLO: <10.000µs 📉 -38.0%) vs baseline: +1.3% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% ✅ ospathjoin_noaspectTime: ✅ 6.233µs (SLO: <10.000µs 📉 -37.7%) vs baseline: -0.3% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.8% ✅ ospathnormcase_aspectTime: ✅ 3.503µs (SLO: <10.000µs 📉 -65.0%) vs baseline: +0.4% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.7% ✅ ospathnormcase_noaspectTime: ✅ 3.565µs (SLO: <10.000µs 📉 -64.4%) vs baseline: +0.3% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.1% ✅ ospathsplit_aspectTime: ✅ 4.830µs (SLO: <10.000µs 📉 -51.7%) vs baseline: -0.5% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ ospathsplit_noaspectTime: ✅ 4.954µs (SLO: <10.000µs 📉 -50.5%) vs baseline: +0.2% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.9% ✅ ospathsplitdrive_aspectTime: ✅ 3.775µs (SLO: <10.000µs 📉 -62.3%) vs baseline: +0.5% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +5.0% ✅ ospathsplitdrive_noaspectTime: ✅ 0.746µs (SLO: <10.000µs 📉 -92.5%) vs baseline: -1.1% Memory: ✅ 41.504MB (SLO: <43.500MB -4.6%) vs baseline: +5.3% ✅ ospathsplitext_aspectTime: ✅ 4.660µs (SLO: <10.000µs 📉 -53.4%) vs baseline: +1.0% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +5.1% ✅ ospathsplitext_noaspectTime: ✅ 4.689µs (SLO: <10.000µs 📉 -53.1%) vs baseline: +0.6% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.445µs (SLO: <20.000µs 📉 -82.8%) vs baseline: 📈 +11.5% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ 1-count-metrics-100-timesTime: ✅ 200.760µs (SLO: <220.000µs -8.7%) vs baseline: +0.1% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 1-distribution-metric-1-timesTime: ✅ 3.456µs (SLO: <20.000µs 📉 -82.7%) vs baseline: -0.2% Memory: ✅ 35.055MB (SLO: <35.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 217.571µs (SLO: <230.000µs -5.4%) vs baseline: -0.5% Memory: ✅ 35.232MB (SLO: <35.500MB 🟡 -0.8%) vs baseline: +4.8% ✅ 1-gauge-metric-1-timesTime: ✅ 2.282µs (SLO: <20.000µs 📉 -88.6%) vs baseline: +0.2% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ 1-gauge-metrics-100-timesTime: ✅ 137.784µs (SLO: <150.000µs -8.1%) vs baseline: +0.4% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.3% ✅ 1-rate-metric-1-timesTime: ✅ 3.226µs (SLO: <20.000µs 📉 -83.9%) vs baseline: -1.1% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.6% ✅ 1-rate-metrics-100-timesTime: ✅ 214.129µs (SLO: <250.000µs 📉 -14.3%) vs baseline: -0.3% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.9% ✅ 100-count-metrics-100-timesTime: ✅ 20.378ms (SLO: <22.000ms -7.4%) vs baseline: +0.2% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.9% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.265ms (SLO: <2.550ms 📉 -11.2%) vs baseline: ~same Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.412ms (SLO: <1.550ms -8.9%) vs baseline: -1.0% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.4% ✅ 100-rate-metrics-100-timesTime: ✅ 2.216ms (SLO: <2.550ms 📉 -13.1%) vs baseline: -0.5% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ flush-1-metricTime: ✅ 4.606µs (SLO: <20.000µs 📉 -77.0%) vs baseline: -0.8% Memory: ✅ 35.173MB (SLO: <35.500MB 🟡 -0.9%) vs baseline: +5.0% ✅ flush-100-metricsTime: ✅ 176.960µs (SLO: <250.000µs 📉 -29.2%) vs baseline: +0.6% Memory: ✅ 35.232MB (SLO: <35.500MB 🟡 -0.8%) vs baseline: +4.7% ✅ flush-1000-metricsTime: ✅ 2.173ms (SLO: <2.500ms 📉 -13.1%) vs baseline: ~same Memory: ✅ 35.979MB (SLO: <36.500MB 🟡 -1.4%) vs baseline: +4.6% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
|
I think backporting this was a mistake, or it wasn't properly backported, the test started being flaky after the backport... |
Backport e327013 from #15860 to 4.1.
Description
This PR unflakes
test_gunicornafter #15814 made it flaky. The problem is that #15814 shifted the line we wanted to see in profiles by 4 lines, making the expected location much rarer.Note this is broken, fixed by #15942