-
Notifications
You must be signed in to change notification settings - Fork 477
test(profiling): skip broken test test_gunicorn.py
#16106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test(profiling): skip broken test test_gunicorn.py
#16106
Conversation
|
|
test_gunicorn, again [backport 4.1]"test_gunicorn, again [backport 4.1]"
|
@KowalskiThomas This was needed to unblock #16088 |
|
/approve |
|
|
Isn't there a way to fix this without breaking back the test? I think the test was very broken before. |
|
@KowalskiThomas hmm, why don't we do that on main and just mark this as skip then? |
test_gunicorn, again [backport 4.1]"test_gunicorn.py
|
@taegyunkim Sorry did you change what the PR does? I'm fine with skipping on 4.1. The test works on main, there were issues with the backport due to some other changes (not backported) changing line numbers. |
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 247 ± 2 ms. The average import time from base is: 249 ± 2 ms. The import time difference between this PR and base is: -2.47 ± 0.08 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate revert-15942-kowalski/test-profiling-unflake-test_gunicorn-again (c302a47) with baseline 4.1 (abbcb50) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 104.195µs (SLO: <130.000µs 📉 -19.8%) vs baseline: +3.2% Memory: ✅ 42.959MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.8% ✅ add_inplace_aspectTime: ✅ 101.611µs (SLO: <130.000µs 📉 -21.8%) vs baseline: +0.4% Memory: ✅ 42.861MB (SLO: <43.250MB 🟡 -0.9%) vs baseline: +5.0% ✅ add_inplace_noaspectTime: ✅ 28.207µs (SLO: <40.000µs 📉 -29.5%) vs baseline: +0.2% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ add_noaspectTime: ✅ 48.826µs (SLO: <70.000µs 📉 -30.2%) vs baseline: -1.5% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.4% ✅ bytearray_aspectTime: ✅ 258.042µs (SLO: <400.000µs 📉 -35.5%) vs baseline: -0.5% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ bytearray_extend_aspectTime: ✅ 644.725µs (SLO: <800.000µs 📉 -19.4%) vs baseline: -0.3% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ bytearray_extend_noaspectTime: ✅ 259.059µs (SLO: <400.000µs 📉 -35.2%) vs baseline: +0.3% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ bytearray_noaspectTime: ✅ 140.001µs (SLO: <300.000µs 📉 -53.3%) vs baseline: +2.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ bytes_aspectTime: ✅ 221.970µs (SLO: <300.000µs 📉 -26.0%) vs baseline: -0.3% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.1% ✅ bytes_noaspectTime: ✅ 133.229µs (SLO: <200.000µs 📉 -33.4%) vs baseline: +1.0% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.7% ✅ bytesio_aspectTime: ✅ 3.890ms (SLO: <5.000ms 📉 -22.2%) vs baseline: +0.7% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ bytesio_noaspectTime: ✅ 316.258µs (SLO: <420.000µs 📉 -24.7%) vs baseline: ~same Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ capitalize_aspectTime: ✅ 88.695µs (SLO: <300.000µs 📉 -70.4%) vs baseline: -0.9% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.4% ✅ capitalize_noaspectTime: ✅ 252.972µs (SLO: <300.000µs 📉 -15.7%) vs baseline: +1.7% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ casefold_aspectTime: ✅ 89.324µs (SLO: <500.000µs 📉 -82.1%) vs baseline: -0.7% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.3% ✅ casefold_noaspectTime: ✅ 302.590µs (SLO: <500.000µs 📉 -39.5%) vs baseline: -0.7% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.6% ✅ decode_aspectTime: ✅ 87.664µs (SLO: <100.000µs 📉 -12.3%) vs baseline: +0.2% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.8% ✅ decode_noaspectTime: ✅ 152.816µs (SLO: <210.000µs 📉 -27.2%) vs baseline: -0.5% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ encode_aspectTime: ✅ 85.295µs (SLO: <200.000µs 📉 -57.4%) vs baseline: -0.2% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.7% ✅ encode_noaspectTime: ✅ 139.817µs (SLO: <200.000µs 📉 -30.1%) vs baseline: -1.9% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ format_aspectTime: ✅ 14.713ms (SLO: <19.200ms 📉 -23.4%) vs baseline: ~same Memory: ✅ 42.861MB (SLO: <43.250MB 🟡 -0.9%) vs baseline: +4.2% ✅ format_map_aspectTime: ✅ 16.523ms (SLO: <21.500ms 📉 -23.2%) vs baseline: +0.1% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.3% ✅ format_map_noaspectTime: ✅ 365.122µs (SLO: <500.000µs 📉 -27.0%) vs baseline: +0.1% Memory: ✅ 42.703MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.1% ✅ format_noaspectTime: ✅ 310.452µs (SLO: <500.000µs 📉 -37.9%) vs baseline: +1.0% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.5% ✅ index_aspectTime: ✅ 145.147µs (SLO: <300.000µs 📉 -51.6%) vs baseline: 📈 +16.4% Memory: ✅ 42.861MB (SLO: <43.250MB 🟡 -0.9%) vs baseline: +4.6% ✅ index_noaspectTime: ✅ 40.392µs (SLO: <300.000µs 📉 -86.5%) vs baseline: -0.4% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +3.9% ✅ join_aspectTime: ✅ 209.942µs (SLO: <300.000µs 📉 -30.0%) vs baseline: +0.6% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ join_noaspectTime: ✅ 142.853µs (SLO: <300.000µs 📉 -52.4%) vs baseline: -0.4% Memory: ✅ 42.762MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.7% ✅ ljust_aspectTime: ✅ 498.117µs (SLO: <700.000µs 📉 -28.8%) vs baseline: -0.6% Memory: ✅ 43.018MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +5.0% ✅ ljust_noaspectTime: ✅ 285.242µs (SLO: <300.000µs -4.9%) vs baseline: 📈 +11.8% Memory: ✅ 42.703MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +4.4% ✅ lower_aspectTime: ✅ 302.121µs (SLO: <500.000µs 📉 -39.6%) vs baseline: -0.4% Memory: ✅ 42.900MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.5% ✅ lower_noaspectTime: ✅ 229.982µs (SLO: <300.000µs 📉 -23.3%) vs baseline: -0.7% Memory: ✅ 42.762MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.6% ✅ lstrip_aspectTime: ✅ 0.267ms (SLO: <3.000ms 📉 -91.1%) vs baseline: -1.3% Memory: ✅ 42.782MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.2% ✅ lstrip_noaspectTime: ✅ 0.178ms (SLO: <3.000ms 📉 -94.1%) vs baseline: +1.0% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.7% ✅ modulo_aspectTime: ✅ 14.372ms (SLO: <18.750ms 📉 -23.4%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.6% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 14.936ms (SLO: <19.350ms 📉 -22.8%) vs baseline: +0.5% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ modulo_aspect_for_bytesTime: ✅ 14.472ms (SLO: <18.900ms 📉 -23.4%) vs baseline: -0.1% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.1% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 14.802ms (SLO: <19.150ms 📉 -22.7%) vs baseline: +0.4% Memory: ✅ 42.841MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.4% ✅ modulo_noaspectTime: ✅ 0.361ms (SLO: <3.000ms 📉 -88.0%) vs baseline: -0.3% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ replace_aspectTime: ✅ 18.704ms (SLO: <24.000ms 📉 -22.1%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <44.000MB -2.4%) vs baseline: +4.4% ✅ replace_noaspectTime: ✅ 512.760µs (SLO: <800.000µs 📉 -35.9%) vs baseline: -0.1% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.6% ✅ repr_aspectTime: ✅ 317.158µs (SLO: <420.000µs 📉 -24.5%) vs baseline: +0.7% Memory: ✅ 42.880MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.6% ✅ repr_noaspectTime: ✅ 46.673µs (SLO: <90.000µs 📉 -48.1%) vs baseline: +0.5% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.2% ✅ rstrip_aspectTime: ✅ 381.104µs (SLO: <500.000µs 📉 -23.8%) vs baseline: +0.4% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ rstrip_noaspectTime: ✅ 178.430µs (SLO: <300.000µs 📉 -40.5%) vs baseline: -1.7% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.0% ✅ slice_aspectTime: ✅ 188.622µs (SLO: <300.000µs 📉 -37.1%) vs baseline: -0.6% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.1% ✅ slice_noaspectTime: ✅ 54.085µs (SLO: <90.000µs 📉 -39.9%) vs baseline: +0.4% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.9% ✅ stringio_aspectTime: ✅ 4.519ms (SLO: <5.000ms -9.6%) vs baseline: 📈 +16.1% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 345.604µs (SLO: <500.000µs 📉 -30.9%) vs baseline: -0.9% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ strip_aspectTime: ✅ 268.872µs (SLO: <350.000µs 📉 -23.2%) vs baseline: -0.3% Memory: ✅ 43.077MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 176.536µs (SLO: <240.000µs 📉 -26.4%) vs baseline: +0.3% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ swapcase_aspectTime: ✅ 338.418µs (SLO: <500.000µs 📉 -32.3%) vs baseline: -1.1% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ swapcase_noaspectTime: ✅ 267.922µs (SLO: <400.000µs 📉 -33.0%) vs baseline: ~same Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ title_aspectTime: ✅ 332.239µs (SLO: <500.000µs 📉 -33.6%) vs baseline: +0.6% Memory: ✅ 42.998MB (SLO: <43.000MB 🟡 ~same) vs baseline: +4.9% ✅ title_noaspectTime: ✅ 257.649µs (SLO: <400.000µs 📉 -35.6%) vs baseline: +0.4% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ translate_aspectTime: ✅ 496.399µs (SLO: <700.000µs 📉 -29.1%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ translate_noaspectTime: ✅ 415.355µs (SLO: <500.000µs 📉 -16.9%) vs baseline: -0.8% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ upper_aspectTime: ✅ 299.349µs (SLO: <500.000µs 📉 -40.1%) vs baseline: -1.2% Memory: ✅ 43.077MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +4.9% ✅ upper_noaspectTime: ✅ 230.220µs (SLO: <400.000µs 📉 -42.4%) vs baseline: -1.0% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.2% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 506.577µs (SLO: <700.000µs 📉 -27.6%) vs baseline: 📈 +21.0% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.6% ✅ ospathbasename_noaspectTime: ✅ 425.678µs (SLO: <700.000µs 📉 -39.2%) vs baseline: -1.1% Memory: ✅ 41.524MB (SLO: <43.500MB -4.5%) vs baseline: +5.1% ✅ ospathjoin_aspectTime: ✅ 610.947µs (SLO: <700.000µs 📉 -12.7%) vs baseline: -0.2% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +5.1% ✅ ospathjoin_noaspectTime: ✅ 619.057µs (SLO: <700.000µs 📉 -11.6%) vs baseline: ~same Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% ✅ ospathnormcase_aspectTime: ✅ 347.582µs (SLO: <700.000µs 📉 -50.3%) vs baseline: -0.4% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +5.1% ✅ ospathnormcase_noaspectTime: ✅ 357.679µs (SLO: <700.000µs 📉 -48.9%) vs baseline: +0.5% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% ✅ ospathsplit_aspectTime: ✅ 481.497µs (SLO: <700.000µs 📉 -31.2%) vs baseline: ~same Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathsplit_noaspectTime: ✅ 491.418µs (SLO: <700.000µs 📉 -29.8%) vs baseline: -0.4% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathsplitdrive_aspectTime: ✅ 373.438µs (SLO: <700.000µs 📉 -46.7%) vs baseline: +0.6% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +4.8% ✅ ospathsplitdrive_noaspectTime: ✅ 73.001µs (SLO: <700.000µs 📉 -89.6%) vs baseline: ~same Memory: ✅ 41.347MB (SLO: <43.500MB -5.0%) vs baseline: +4.6% ✅ ospathsplitext_aspectTime: ✅ 459.837µs (SLO: <700.000µs 📉 -34.3%) vs baseline: ~same Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ ospathsplitext_noaspectTime: ✅ 471.242µs (SLO: <700.000µs 📉 -32.7%) vs baseline: +1.1% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.338µs (SLO: <20.000µs 📉 -83.3%) vs baseline: 📈 +11.3% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 1-count-metrics-100-timesTime: ✅ 202.273µs (SLO: <220.000µs -8.1%) vs baseline: -1.1% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 1-distribution-metric-1-timesTime: ✅ 3.371µs (SLO: <20.000µs 📉 -83.1%) vs baseline: -0.5% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.8% ✅ 1-distribution-metrics-100-timesTime: ✅ 218.249µs (SLO: <230.000µs -5.1%) vs baseline: +0.2% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ 1-gauge-metric-1-timesTime: ✅ 2.199µs (SLO: <20.000µs 📉 -89.0%) vs baseline: +0.4% Memory: ✅ 34.741MB (SLO: <35.500MB -2.1%) vs baseline: +4.3% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.630µs (SLO: <150.000µs -8.9%) vs baseline: +0.3% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.2% ✅ 1-rate-metric-1-timesTime: ✅ 3.166µs (SLO: <20.000µs 📉 -84.2%) vs baseline: +0.2% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.5% ✅ 1-rate-metrics-100-timesTime: ✅ 215.975µs (SLO: <250.000µs 📉 -13.6%) vs baseline: -1.1% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 100-count-metrics-100-timesTime: ✅ 20.399ms (SLO: <22.000ms -7.3%) vs baseline: +1.0% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.280ms (SLO: <2.550ms 📉 -10.6%) vs baseline: +0.8% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.397ms (SLO: <1.550ms -9.9%) vs baseline: +0.3% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.8% ✅ 100-rate-metrics-100-timesTime: ✅ 2.206ms (SLO: <2.550ms 📉 -13.5%) vs baseline: -0.5% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.1% ✅ flush-1-metricTime: ✅ 4.625µs (SLO: <20.000µs 📉 -76.9%) vs baseline: +0.2% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ flush-100-metricsTime: ✅ 174.649µs (SLO: <250.000µs 📉 -30.1%) vs baseline: -0.4% Memory: ✅ 35.193MB (SLO: <35.500MB 🟡 -0.9%) vs baseline: +4.6% ✅ flush-1000-metricsTime: ✅ 2.185ms (SLO: <2.500ms 📉 -12.6%) vs baseline: +0.4% Memory: ✅ 36.078MB (SLO: <36.500MB 🟡 -1.2%) vs baseline: +5.0% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
Reverts #15942