Add benchmarks for concurrent range deletions in Text and Tree#1214
Merged
Conversation
WalkthroughA new benchmark function, Changes
Sequence Diagram(s)sequenceDiagram
participant TR as Test Runner
participant BD as BenchmarkDocumentDeletion
participant HF as Helper Functions
TR->>BD: Invoke benchmark
BD->>HF: Run single deletion benchmarks (text & tree - all & range)
BD->>HF: Run concurrent deletion benchmarks (text & tree - range & edit delete all)
HF-->>BD: Return timing metrics
BD-->>TR: Report benchmark results
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
hackerwins-yorkie
left a comment
hackerwins-yorkie
left a comment
There was a problem hiding this comment.
Go Benchmark Analysis 📊
This is a comparison result between the previous(8dcf51e) and the current commit(c0e119e).
Significant Changes (≥20% difference)
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| BenchmarkDocument/nested_update_test/ (ns/op) | 21114.00 ns | 16663.00 ns | 🟢 -21.08% |
Key Observations 🔍
- Significant improvement (🟢 -21.08%) in the
BenchmarkDocument/nested_update_testsuite, with a notable decrease in execution time from 21,114 ns/op to 16,663 ns/op. - The benchmark suites across various categories generally exhibit relatively stable performance, with most changes falling within a reasonable range, indicating consistent performance.
Overall, the benchmark data shows satisfactory performance across the board, with no alarming deviations or trends present.
Detailed Test Results
BenchmarkDocument
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| constructor_test/ (ns/op) | 1477.00 ns | 1467.00 ns | 🟢 -0.68% |
| constructor_test/ (B/op) | 1.43 KB | 1.43 KB | ⚪ 0% |
| constructor_test/ (allocs/op) | 25 allocs | 25 allocs | ⚪ 0% |
| status_test/ (ns/op) | 1064.00 ns | 1061.00 ns | 🟢 -0.28% |
| status_test/ (B/op) | 1.40 KB | 1.40 KB | ⚪ 0% |
| status_test/ (allocs/op) | 23 allocs | 23 allocs | ⚪ 0% |
| equals_test/ (ns/op) | 7954.00 ns | 7878.00 ns | 🟢 -0.96% |
| equals_test/ (B/op) | 7.71 KB | 7.71 KB | ⚪ 0% |
| equals_test/ (allocs/op) | 132 allocs | 132 allocs | ⚪ 0% |
| nested_update_test/ (ns/op) | 21114.00 ns | 16663.00 ns | 🟢 -21.08% |
| nested_update_test/ (B/op) | 12.36 KB | 12.36 KB | ⚪ 0% |
| nested_update_test/ (allocs/op) | 259 allocs | 259 allocs | ⚪ 0% |
| delete_test/ (ns/op) | 22692.00 ns | 22649.00 ns | 🟢 -0.19% |
| delete_test/ (B/op) | 15.84 KB | 15.84 KB | ⚪ 0% |
| delete_test/ (allocs/op) | 340 allocs | 340 allocs | ⚪ 0% |
| object_test/ (ns/op) | 8535.00 ns | 9722.00 ns | 🔴 +13.91% |
| object_test/ (B/op) | 7.08 KB | 7.08 KB | ⚪ 0% |
| object_test/ (allocs/op) | 119 allocs | 119 allocs | ⚪ 0% |
| array_test/ (ns/op) | 28174.00 ns | 28207.00 ns | 🔴 +0.12% |
| array_test/ (B/op) | 12.19 KB | 12.19 KB | ⚪ 0% |
| array_test/ (allocs/op) | 274 allocs | 274 allocs | ⚪ 0% |
| text_test/ (ns/op) | 31655.00 ns | 31770.00 ns | 🔴 +0.36% |
| text_test/ (B/op) | 15.24 KB | 15.24 KB | ⚪ 0% |
| text_test/ (allocs/op) | 485 allocs | 485 allocs | ⚪ 0% |
| text_composition_test/ (ns/op) | 32127.00 ns | 32142.00 ns | 🔴 +0.05% |
| text_composition_test/ (B/op) | 18.75 KB | 18.75 KB | ⚪ 0% |
| text_composition_test/ (allocs/op) | 502 allocs | 502 allocs | ⚪ 0% |
| rich_text_test/ (ns/op) | 86482.00 ns | 85997.00 ns | 🟢 -0.56% |
| rich_text_test/ (B/op) | 39.40 KB | 39.40 KB | 🟢 -0.02% |
| rich_text_test/ (allocs/op) | 1,147 allocs | 1,147 allocs | ⚪ 0% |
| counter_test/ (ns/op) | 18048.00 ns | 18460.00 ns | 🔴 +2.28% |
| counter_test/ (B/op) | 11.86 KB | 11.86 KB | ⚪ 0% |
| counter_test/ (allocs/op) | 254 allocs | 254 allocs | ⚪ 0% |
| text_edit_gc_100/ (ns/op) | 1.41 ms | 1.40 ms | 🟢 -0.24% |
| text_edit_gc_100/ (B/op) | 864.95 KB | 864.91 KB | ⚪ 0% |
| text_edit_gc_100/ (allocs/op) | 17,282 allocs | 17,282 allocs | ⚪ 0% |
| text_edit_gc_1000/ (ns/op) | 52.48 ms | 52.99 ms | 🔴 +0.97% |
| text_edit_gc_1000/ (B/op) | 46.84 MB | 46.84 MB | ⚪ 0% |
| text_edit_gc_1000/ (allocs/op) | 185,594 allocs | 185,583 allocs | ⚪ 0% |
| text_split_gc_100/ (ns/op) | 2.14 ms | 2.13 ms | 🟢 -0.12% |
| text_split_gc_100/ (B/op) | 1.58 MB | 1.58 MB | ⚪ 0% |
| text_split_gc_100/ (allocs/op) | 15,952 allocs | 15,951 allocs | ⚪ 0% |
| text_split_gc_1000/ (ns/op) | 128.40 ms | 128.51 ms | 🔴 +0.08% |
| text_split_gc_1000/ (B/op) | 137.79 MB | 137.79 MB | ⚪ 0% |
| text_split_gc_1000/ (allocs/op) | 185,008 allocs | 185,002 allocs | ⚪ 0% |
| text_delete_all_10000/ (ns/op) | 16.83 ms | N/A | 🗑️ Removed |
| text_delete_all_10000/ (B/op) | 10.58 MB | N/A | 🗑️ Removed |
| text_delete_all_10000/ (allocs/op) | 56,136 allocs | N/A | 🗑️ Removed |
| text_delete_all_100000/ (ns/op) | 291.44 ms | N/A | 🗑️ Removed |
| text_delete_all_100000/ (B/op) | 105.51 MB | N/A | 🗑️ Removed |
| text_delete_all_100000/ (allocs/op) | 566,023 allocs | N/A | 🗑️ Removed |
| text_100/ (ns/op) | 233738.00 ns | 233847.00 ns | 🔴 +0.05% |
| text_100/ (B/op) | 120.99 KB | 120.95 KB | 🟢 -0.03% |
| text_100/ (allocs/op) | 5,182 allocs | 5,182 allocs | ⚪ 0% |
| text_1000/ (ns/op) | 2.48 ms | 2.51 ms | 🔴 +1.20% |
| text_1000/ (B/op) | 1.16 MB | 1.16 MB | ⚪ 0% |
| text_1000/ (allocs/op) | 51,085 allocs | 51,085 allocs | ⚪ 0% |
| array_1000/ (ns/op) | 1.23 ms | 1.24 ms | 🔴 +1.16% |
| array_1000/ (B/op) | 1.09 MB | 1.09 MB | ⚪ 0% |
| array_1000/ (allocs/op) | 11,880 allocs | 11,880 allocs | ⚪ 0% |
| array_10000/ (ns/op) | 13.53 ms | 13.47 ms | 🟢 -0.47% |
| array_10000/ (B/op) | 9.89 MB | 9.89 MB | 🔴 +0.01% |
| array_10000/ (allocs/op) | 120,730 allocs | 120,734 allocs | ⚪ 0% |
| array_gc_100/ (ns/op) | 131805.00 ns | 131674.00 ns | 🟢 -0.10% |
| array_gc_100/ (B/op) | 99.94 KB | 99.94 KB | ⚪ 0% |
| array_gc_100/ (allocs/op) | 1,267 allocs | 1,267 allocs | ⚪ 0% |
| array_gc_1000/ (ns/op) | 1.42 ms | 1.43 ms | 🔴 +0.70% |
| array_gc_1000/ (B/op) | 1.14 MB | 1.14 MB | ⚪ 0% |
| array_gc_1000/ (allocs/op) | 12,927 allocs | 12,927 allocs | ⚪ 0% |
| counter_1000/ (ns/op) | 201610.00 ns | 199991.00 ns | 🟢 -0.80% |
| counter_1000/ (B/op) | 178.19 KB | 178.18 KB | ⚪ 0% |
| counter_1000/ (allocs/op) | 5,772 allocs | 5,772 allocs | ⚪ 0% |
| counter_10000/ (ns/op) | 2.16 ms | 2.14 ms | 🟢 -1.03% |
| counter_10000/ (B/op) | 2.07 MB | 2.07 MB | ⚪ 0% |
| counter_10000/ (allocs/op) | 59,779 allocs | 59,779 allocs | ⚪ 0% |
| object_1000/ (ns/op) | 1.40 ms | 1.40 ms | 🟢 -0.21% |
| object_1000/ (B/op) | 1.44 MB | 1.44 MB | ⚪ 0% |
| object_1000/ (allocs/op) | 9,925 allocs | 9,926 allocs | 🔴 +0.01% |
| object_10000/ (ns/op) | 14.60 ms | 14.21 ms | 🟢 -2.64% |
| object_10000/ (B/op) | 12.35 MB | 12.35 MB | ⚪ 0% |
| object_10000/ (allocs/op) | 101,228 allocs | 101,229 allocs | ⚪ 0% |
| tree_100/ (ns/op) | 1.03 ms | 1.03 ms | 🔴 +0.12% |
| tree_100/ (B/op) | 951.07 KB | 951.08 KB | ⚪ 0% |
| tree_100/ (allocs/op) | 6,103 allocs | 6,103 allocs | ⚪ 0% |
| tree_1000/ (ns/op) | 77.09 ms | 76.43 ms | 🟢 -0.87% |
| tree_1000/ (B/op) | 86.58 MB | 86.58 MB | ⚪ 0% |
| tree_1000/ (allocs/op) | 60,112 allocs | 60,112 allocs | ⚪ 0% |
| tree_10000/ (ns/op) | 9.55 s | 9.62 s | 🔴 +0.78% |
| tree_10000/ (B/op) | 8.58 GB | 8.58 GB | ⚪ 0% |
| tree_10000/ (allocs/op) | 600,199 allocs | 600,187 allocs | ⚪ 0% |
| tree_delete_all_1000/ (ns/op) | 77.05 ms | N/A | 🗑️ Removed |
| tree_delete_all_1000/ (B/op) | 87.57 MB | N/A | 🗑️ Removed |
| tree_delete_all_1000/ (allocs/op) | 75,290 allocs | N/A | 🗑️ Removed |
| tree_edit_gc_100/ (ns/op) | 3.83 ms | 3.83 ms | 🟢 -0.03% |
| tree_edit_gc_100/ (B/op) | 4.15 MB | 4.15 MB | ⚪ 0% |
| tree_edit_gc_100/ (allocs/op) | 15,147 allocs | 15,147 allocs | ⚪ 0% |
| tree_edit_gc_1000/ (ns/op) | 318.09 ms | 312.34 ms | 🟢 -1.81% |
| tree_edit_gc_1000/ (B/op) | 384.05 MB | 384.04 MB | ⚪ 0% |
| tree_edit_gc_1000/ (allocs/op) | 154,953 allocs | 154,929 allocs | 🟢 -0.02% |
| tree_split_gc_100/ (ns/op) | 2.64 ms | 2.62 ms | 🟢 -0.68% |
| tree_split_gc_100/ (B/op) | 2.41 MB | 2.41 MB | ⚪ 0% |
| tree_split_gc_100/ (allocs/op) | 11,132 allocs | 11,132 allocs | ⚪ 0% |
| tree_split_gc_1000/ (ns/op) | 191.19 ms | 192.15 ms | 🔴 +0.50% |
| tree_split_gc_1000/ (B/op) | 222.50 MB | 222.50 MB | ⚪ 0% |
| tree_split_gc_1000/ (allocs/op) | 122,062 allocs | 122,054 allocs | ⚪ 0% |
BenchmarkRPC
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| client_to_server/ (ns/op) | 445.24 ms | 446.94 ms | 🔴 +0.38% |
| client_to_server/ (B/op) | 17.39 MB | 16.72 MB | 🟢 -3.85% |
| client_to_server/ (allocs/op) | 223,481 allocs | 223,613 allocs | 🔴 +0.06% |
| client_to_client_via_server/ (ns/op) | 571.93 ms | 578.32 ms | 🔴 +1.12% |
| client_to_client_via_server/ (B/op) | 25.83 MB | 27.92 MB | 🔴 +8.07% |
| client_to_client_via_server/ (allocs/op) | 340,104 allocs | 340,315 allocs | 🔴 +0.06% |
| attach_large_document/ (ns/op) | 1.29 s | 1.24 s | 🟢 -4.11% |
| attach_large_document/ (B/op) | 1.91 GB | 1.92 GB | 🔴 +0.55% |
| attach_large_document/ (allocs/op) | 12,454 allocs | 12,236 allocs | 🟢 -1.75% |
| adminCli_to_server/ (ns/op) | 558.89 ms | 561.46 ms | 🔴 +0.46% |
| adminCli_to_server/ (B/op) | 22.31 MB | 21.50 MB | 🟢 -3.62% |
| adminCli_to_server/ (allocs/op) | 316,681 allocs | 316,657 allocs | ⚪ 0% |
BenchmarkLocker
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| (ns/op) | 86.14 ns | 85.29 ns | 🟢 -0.99% |
| (B/op) | 32.00 B | 32.00 B | ⚪ 0% |
| (allocs/op) | 1 allocs | 1 allocs | ⚪ 0% |
BenchmarkLockerParallel
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| (ns/op) | 45.22 ns | 45.23 ns | 🔴 +0.02% |
| (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
BenchmarkLockerMoreKeys
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| (ns/op) | 185.80 ns | 184.90 ns | 🟢 -0.48% |
| (B/op) | 31.00 B | 30.00 B | 🟢 -3.23% |
| (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
BenchmarkRWLocker
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| RWLock_rate_2/ (ns/op) | 49.60 ns | 49.45 ns | 🟢 -0.30% |
| RWLock_rate_2/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| RWLock_rate_2/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| RWLock_rate_10/ (ns/op) | 44.47 ns | 43.67 ns | 🟢 -1.80% |
| RWLock_rate_10/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| RWLock_rate_10/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| RWLock_rate_100/ (ns/op) | 61.17 ns | 59.93 ns | 🟢 -2.03% |
| RWLock_rate_100/ (B/op) | 2.00 B | 2.00 B | ⚪ 0% |
| RWLock_rate_100/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| RWLock_rate_1000/ (ns/op) | 92.37 ns | 91.02 ns | 🟢 -1.46% |
| RWLock_rate_1000/ (B/op) | 8.00 B | 8.00 B | ⚪ 0% |
| RWLock_rate_1000/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
BenchmarkChange
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| Push_10_Changes/ (ns/op) | 5.05 ms | 5.01 ms | 🟢 -0.72% |
| Push_10_Changes/ (B/op) | 149.08 KB | 149.23 KB | 🔴 +0.10% |
| Push_10_Changes/ (allocs/op) | 1,625 allocs | 1,625 allocs | ⚪ 0% |
| Push_100_Changes/ (ns/op) | 18.64 ms | 19.29 ms | 🔴 +3.46% |
| Push_100_Changes/ (B/op) | 780.23 KB | 759.96 KB | 🟢 -2.60% |
| Push_100_Changes/ (allocs/op) | 8,514 allocs | 8,510 allocs | 🟢 -0.05% |
| Push_1000_Changes/ (ns/op) | 150.17 ms | 150.91 ms | 🔴 +0.49% |
| Push_1000_Changes/ (B/op) | 7.15 MB | 7.20 MB | 🔴 +0.77% |
| Push_1000_Changes/ (allocs/op) | 79,331 allocs | 79,327 allocs | ⚪ 0% |
| Pull_10_Changes/ (ns/op) | 4.08 ms | 4.10 ms | 🔴 +0.51% |
| Pull_10_Changes/ (B/op) | 122.88 KB | 122.86 KB | 🟢 -0.02% |
| Pull_10_Changes/ (allocs/op) | 1,454 allocs | 1,454 allocs | ⚪ 0% |
| Pull_100_Changes/ (ns/op) | 5.86 ms | 5.99 ms | 🔴 +2.14% |
| Pull_100_Changes/ (B/op) | 351.03 KB | 351.04 KB | ⚪ 0% |
| Pull_100_Changes/ (allocs/op) | 5,177 allocs | 5,179 allocs | 🔴 +0.04% |
| Pull_1000_Changes/ (ns/op) | 11.50 ms | 11.62 ms | 🔴 +1.06% |
| Pull_1000_Changes/ (B/op) | 2.19 MB | 2.19 MB | 🟢 -0.19% |
| Pull_1000_Changes/ (allocs/op) | 44,680 allocs | 44,680 allocs | ⚪ 0% |
BenchmarkSnapshot
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| Push_3KB_snapshot/ (ns/op) | 21.53 ms | 21.46 ms | 🟢 -0.30% |
| Push_3KB_snapshot/ (B/op) | 891.18 KB | 902.46 KB | 🔴 +1.27% |
| Push_3KB_snapshot/ (allocs/op) | 8,517 allocs | 8,515 allocs | 🟢 -0.02% |
| Push_30KB_snapshot/ (ns/op) | 153.69 ms | 154.44 ms | 🔴 +0.49% |
| Push_30KB_snapshot/ (B/op) | 7.51 MB | 7.57 MB | 🔴 +0.71% |
| Push_30KB_snapshot/ (allocs/op) | 81,361 allocs | 81,552 allocs | 🔴 +0.23% |
| Pull_3KB_snapshot/ (ns/op) | 8.27 ms | 8.20 ms | 🟢 -0.79% |
| Pull_3KB_snapshot/ (B/op) | 1.05 MB | 1.05 MB | 🟢 -0.07% |
| Pull_3KB_snapshot/ (allocs/op) | 19,243 allocs | 19,244 allocs | ⚪ 0% |
| Pull_30KB_snapshot/ (ns/op) | 20.38 ms | 19.82 ms | 🟢 -2.73% |
| Pull_30KB_snapshot/ (B/op) | 8.76 MB | 8.75 MB | 🟢 -0.10% |
| Pull_30KB_snapshot/ (allocs/op) | 185,676 allocs | 185,674 allocs | ⚪ 0% |
BenchmarkSplayTree
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| stress_test_100000/ (ns/op) | 0.18 ns | 0.20 ns | 🔴 +9.93% |
| stress_test_100000/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| stress_test_100000/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| stress_test_200000/ (ns/op) | 0.37 ns | 0.38 ns | 🔴 +2.43% |
| stress_test_200000/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| stress_test_200000/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| stress_test_300000/ (ns/op) | 0.57 ns | 0.58 ns | 🔴 +1.22% |
| stress_test_300000/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| stress_test_300000/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| random_access_100000/ (ns/op) | 0.01 ns | 0.01 ns | 🟢 -0.96% |
| random_access_100000/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| random_access_100000/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| random_access_200000/ (ns/op) | 0.03 ns | 0.03 ns | 🟢 -1.59% |
| random_access_200000/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| random_access_200000/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| random_access_300000/ (ns/op) | 0.04 ns | 0.04 ns | 🟢 -8.49% |
| random_access_300000/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| random_access_300000/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
| editing_trace_bench/ (ns/op) | 0.00 ns | 0.00 ns | 🔴 +18.44% |
| editing_trace_bench/ (B/op) | 0.00 B | 0.00 B | ⚪ 0% |
| editing_trace_bench/ (allocs/op) | 0 allocs | 0 allocs | ⚪ 0% |
BenchmarkSync
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| memory_sync_10_test/ (ns/op) | 7198.00 ns | 7273.00 ns | 🔴 +1.04% |
| memory_sync_10_test/ (B/op) | 1.34 KB | 1.34 KB | 🟢 -0.07% |
| memory_sync_10_test/ (allocs/op) | 35 allocs | 35 allocs | ⚪ 0% |
| memory_sync_100_test/ (ns/op) | 55378.00 ns | 54495.00 ns | 🟢 -1.59% |
| memory_sync_100_test/ (B/op) | 9.51 KB | 9.53 KB | 🔴 +0.18% |
| memory_sync_100_test/ (allocs/op) | 268 allocs | 269 allocs | 🔴 +0.37% |
| memory_sync_1000_test/ (ns/op) | 611708.00 ns | 604480.00 ns | 🟢 -1.18% |
| memory_sync_1000_test/ (B/op) | 76.34 KB | 76.23 KB | 🟢 -0.14% |
| memory_sync_1000_test/ (allocs/op) | 2,124 allocs | 2,121 allocs | 🟢 -0.14% |
| memory_sync_10000_test/ (ns/op) | 8.01 ms | 8.29 ms | 🔴 +3.47% |
| memory_sync_10000_test/ (B/op) | 748.62 KB | 763.09 KB | 🔴 +1.93% |
| memory_sync_10000_test/ (allocs/op) | 20,381 allocs | 20,464 allocs | 🔴 +0.41% |
BenchmarkTextEditing
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| (ns/op) | 5.36 s | 5.43 s | 🔴 +1.44% |
| (B/op) | 3.92 GB | 3.92 GB | ⚪ 0% |
| (allocs/op) | 20,620,084 allocs | 20,620,142 allocs | ⚪ 0% |
BenchmarkTree
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| 10000_vertices_to_protobuf/ (ns/op) | 4.32 ms | 4.51 ms | 🔴 +4.38% |
| 10000_vertices_to_protobuf/ (B/op) | 6.36 MB | 6.36 MB | ⚪ 0% |
| 10000_vertices_to_protobuf/ (allocs/op) | 70,025 allocs | 70,025 allocs | ⚪ 0% |
| 10000_vertices_from_protobuf/ (ns/op) | 223.88 ms | 223.03 ms | 🟢 -0.38% |
| 10000_vertices_from_protobuf/ (B/op) | 442.31 MB | 442.31 MB | ⚪ 0% |
| 10000_vertices_from_protobuf/ (allocs/op) | 290,048 allocs | 290,047 allocs | ⚪ 0% |
| 20000_vertices_to_protobuf/ (ns/op) | 9.12 ms | 9.26 ms | 🔴 +1.61% |
| 20000_vertices_to_protobuf/ (B/op) | 12.89 MB | 12.89 MB | ⚪ 0% |
| 20000_vertices_to_protobuf/ (allocs/op) | 140,028 allocs | 140,029 allocs | ⚪ 0% |
| 20000_vertices_from_protobuf/ (ns/op) | 876.73 ms | 886.70 ms | 🔴 +1.14% |
| 20000_vertices_from_protobuf/ (B/op) | 1.70 GB | 1.70 GB | ⚪ 0% |
| 20000_vertices_from_protobuf/ (allocs/op) | 580,123 allocs | 580,075 allocs | ⚪ 0% |
| 30000_vertices_to_protobuf/ (ns/op) | 14.16 ms | 14.54 ms | 🔴 +2.70% |
| 30000_vertices_to_protobuf/ (B/op) | 18.98 MB | 18.98 MB | ⚪ 0% |
| 30000_vertices_to_protobuf/ (allocs/op) | 210,029 allocs | 210,029 allocs | ⚪ 0% |
| 30000_vertices_from_protobuf/ (ns/op) | 2.01 s | 1.99 s | 🟢 -1.04% |
| 30000_vertices_from_protobuf/ (B/op) | 3.75 GB | 3.75 GB | ⚪ 0% |
| 30000_vertices_from_protobuf/ (allocs/op) | 870,132 allocs | 870,129 allocs | ⚪ 0% |
BenchmarkVersionVector
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| clients_10/ (ns/op) | 169.60 ms | 171.33 ms | 🔴 +1.02% |
| clients_10/ (1_changepack(bytes)) | 745.00 B | 745.00 B | ⚪ 0% |
| clients_10/ (2_snapshot(bytes)) | 379.00 B | 379.00 B | ⚪ 0% |
| clients_10/ (3_pushpull(ms)) | 8.00 ms | 8.00 ms | ⚪ 0% |
| clients_10/ (4_attach(ms)) | 6.00 ms | 6.00 ms | ⚪ 0% |
| clients_10/ (5_changepack_after_detach(bytes)) | 805.00 B | 805.00 B | ⚪ 0% |
| clients_10/ (6_snapshot_after_detach(bytes)) | 136.00 B | 136.00 B | ⚪ 0% |
| clients_10/ (7_pushpull_after_detach(ms)) | 8.00 ms | 8.00 ms | ⚪ 0% |
| clients_10/ (B/op) | 21.34 MB | 21.88 MB | 🔴 +2.56% |
| clients_10/ (allocs/op) | 83,280 allocs | 83,300 allocs | 🔴 +0.02% |
| clients_100/ (ns/op) | 1.47 s | 1.49 s | 🔴 +1.43% |
| clients_100/ (1_changepack(bytes)) | 6.14 KB | 6.14 KB | ⚪ 0% |
| clients_100/ (2_snapshot(bytes)) | 3.08 KB | 3.08 KB | ⚪ 0% |
| clients_100/ (3_pushpull(ms)) | 12.00 ms | 12.00 ms | ⚪ 0% |
| clients_100/ (4_attach(ms)) | 9.00 ms | 10.00 ms | 🔴 +11.11% |
| clients_100/ (5_changepack_after_detach(bytes)) | 6.21 KB | 6.21 KB | ⚪ 0% |
| clients_100/ (6_snapshot_after_detach(bytes)) | 137.00 B | 137.00 B | ⚪ 0% |
| clients_100/ (7_pushpull_after_detach(ms)) | 10.00 ms | 10.00 ms | ⚪ 0% |
| clients_100/ (B/op) | 215.21 MB | 216.84 MB | 🔴 +0.76% |
| clients_100/ (allocs/op) | 1,481,507 allocs | 1,481,560 allocs | ⚪ 0% |
| clients_1000/ (ns/op) | 43.39 s | 44.15 s | 🔴 +1.75% |
| clients_1000/ (1_changepack(bytes)) | 60.16 KB | 60.16 KB | ⚪ 0% |
| clients_1000/ (2_snapshot(bytes)) | 30.08 KB | 30.08 KB | ⚪ 0% |
| clients_1000/ (3_pushpull(ms)) | 115.00 ms | 114.00 ms | 🟢 -0.87% |
| clients_1000/ (4_attach(ms)) | 79.00 ms | 72.00 ms | 🟢 -8.86% |
| clients_1000/ (5_changepack_after_detach(bytes)) | 60.22 KB | 60.22 KB | ⚪ 0% |
| clients_1000/ (6_snapshot_after_detach(bytes)) | 139.00 B | 139.00 B | ⚪ 0% |
| clients_1000/ (7_pushpull_after_detach(ms)) | 22.00 ms | 23.00 ms | 🔴 +4.55% |
| clients_1000/ (B/op) | 6.43 GB | 6.43 GB | 🟢 -0.02% |
| clients_1000/ (allocs/op) | 93,356,177 allocs | 93,357,633 allocs | ⚪ 0% |
BenchmarkWebhook
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| Send_10_Webhooks_to_10_Endpoints/ (ns/op) | 12.13 ms | 12.21 ms | 🔴 +0.67% |
| Send_10_Webhooks_to_10_Endpoints/ (B/op) | 809.14 KB | 808.69 KB | 🟢 -0.05% |
| Send_10_Webhooks_to_10_Endpoints/ (allocs/op) | 10,115 allocs | 10,112 allocs | 🟢 -0.03% |
| Send_100_Webhooks_to_10_Endpoints/ (ns/op) | 120.95 ms | 121.23 ms | 🔴 +0.23% |
| Send_100_Webhooks_to_10_Endpoints/ (B/op) | 8.09 MB | 8.09 MB | ⚪ 0% |
| Send_100_Webhooks_to_10_Endpoints/ (allocs/op) | 101,149 allocs | 101,145 allocs | ⚪ 0% |
| Send_10_Webhooks_to_100_Endpoints/ (ns/op) | 124.21 ms | 126.26 ms | 🔴 +1.65% |
| Send_10_Webhooks_to_100_Endpoints/ (B/op) | 8.24 MB | 8.26 MB | 🔴 +0.25% |
| Send_10_Webhooks_to_100_Endpoints/ (allocs/op) | 102,199 allocs | 102,347 allocs | 🔴 +0.14% |
| Send_100_Webhooks_to_100_Endpoints/ (ns/op) | 1.23 s | 1.25 s | 🔴 +1.04% |
| Send_100_Webhooks_to_100_Endpoints/ (B/op) | 82.20 MB | 82.22 MB | 🔴 +0.02% |
| Send_100_Webhooks_to_100_Endpoints/ (allocs/op) | 1,020,877 allocs | 1,020,972 allocs | ⚪ 0% |
| Send_10_Webhooks_to_1000_Endpoints/ (ns/op) | 2.96 s | 2.98 s | 🔴 +0.50% |
| Send_10_Webhooks_to_1000_Endpoints/ (B/op) | 208.24 MB | 208.10 MB | 🟢 -0.06% |
| Send_10_Webhooks_to_1000_Endpoints/ (allocs/op) | 1,702,485 allocs | 1,702,535 allocs | ⚪ 0% |
BenchmarkWebhookWithLimit
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| Send_10_Webhooks_to_10_Endpoints_with_limit/ (ns/op) | 2.48 ms | 2.50 ms | 🔴 +0.88% |
| Send_10_Webhooks_to_10_Endpoints_with_limit/ (B/op) | 179.86 KB | 179.86 KB | ⚪ 0% |
| Send_10_Webhooks_to_10_Endpoints_with_limit/ (allocs/op) | 2,255 allocs | 2,255 allocs | ⚪ 0% |
| Send_100_Webhooks_to_10_Endpoints_with_limit/ (ns/op) | 2.76 ms | 2.77 ms | 🔴 +0.47% |
| Send_100_Webhooks_to_10_Endpoints_with_limit/ (B/op) | 302.38 KB | 302.35 KB | 🟢 -0.01% |
| Send_100_Webhooks_to_10_Endpoints_with_limit/ (allocs/op) | 4,056 allocs | 4,056 allocs | ⚪ 0% |
| Send_10_Webhooks_to_100_Endpoints_with_limit/ (ns/op) | 25.65 ms | 25.71 ms | 🔴 +0.23% |
| Send_10_Webhooks_to_100_Endpoints_with_limit/ (B/op) | 1.83 MB | 1.83 MB | 🔴 +0.03% |
| Send_10_Webhooks_to_100_Endpoints_with_limit/ (allocs/op) | 22,659 allocs | 22,661 allocs | ⚪ 0% |
| Send_100_Webhooks_to_100_Endpoints_with_limit/ (ns/op) | 28.52 ms | 28.49 ms | 🟢 -0.08% |
| Send_100_Webhooks_to_100_Endpoints_with_limit/ (B/op) | 3.12 MB | 3.12 MB | ⚪ 0% |
| Send_100_Webhooks_to_100_Endpoints_with_limit/ (allocs/op) | 40,691 allocs | 40,694 allocs | ⚪ 0% |
| Send_10_Webhooks_to_1000_Endpoints_with_limit/ (ns/op) | 410.14 ms | 413.27 ms | 🔴 +0.76% |
| Send_10_Webhooks_to_1000_Endpoints_with_limit/ (B/op) | 43.89 MB | 43.93 MB | 🔴 +0.10% |
| Send_10_Webhooks_to_1000_Endpoints_with_limit/ (allocs/op) | 375,297 allocs | 375,351 allocs | 🔴 +0.01% |
BenchmarkDocumentDeletion
| Benchmark suite | Previous | Current | Change |
|---|---|---|---|
| single_text_delete_all_10000/ (ns/op) | N/A | 16.77 ms | ✅ New |
| single_text_delete_all_10000/ (B/op) | N/A | 10.58 MB | ✅ New |
| single_text_delete_all_10000/ (allocs/op) | N/A | 56,139 allocs | ✅ New |
| single_text_delete_all_100000/ (ns/op) | N/A | 260.70 ms | ✅ New |
| single_text_delete_all_100000/ (B/op) | N/A | 105.56 MB | ✅ New |
| single_text_delete_all_100000/ (allocs/op) | N/A | 566,131 allocs | ✅ New |
| single_tree_delete_all_1000/ (ns/op) | N/A | 83.00 ms | ✅ New |
| single_tree_delete_all_1000/ (B/op) | N/A | 87.57 MB | ✅ New |
| single_tree_delete_all_1000/ (allocs/op) | N/A | 75,292 allocs | ✅ New |
| single_text_delete_range_100/ (ns/op) | N/A | 444256.00 ns | ✅ New |
| single_text_delete_range_100/ (B/op) | N/A | 255.15 KB | ✅ New |
| single_text_delete_range_100/ (allocs/op) | N/A | 7,112 allocs | ✅ New |
| single_text_delete_range_1000/ (ns/op) | N/A | 8.65 ms | ✅ New |
| single_text_delete_range_1000/ (B/op) | N/A | 6.50 MB | ✅ New |
| single_text_delete_range_1000/ (allocs/op) | N/A | 73,772 allocs | ✅ New |
| single_tree_delete_range_100/ (ns/op) | N/A | 1.36 ms | ✅ New |
| single_tree_delete_range_100/ (B/op) | N/A | 1.19 MB | ✅ New |
| single_tree_delete_range_100/ (allocs/op) | N/A | 8,012 allocs | ✅ New |
| single_tree_delete_range_1000/ (ns/op) | N/A | 93.72 ms | ✅ New |
| single_tree_delete_range_1000/ (B/op) | N/A | 102.37 MB | ✅ New |
| single_tree_delete_range_1000/ (allocs/op) | N/A | 81,923 allocs | ✅ New |
| concurrent_text_delete_range_100/ (ns/op) | N/A | 1.34 s | ✅ New |
| concurrent_text_delete_range_100/ (B/op) | N/A | 98.11 MB | ✅ New |
| concurrent_text_delete_range_100/ (allocs/op) | N/A | 1,219,161 allocs | ✅ New |
| concurrent_text_delete_range_1000/ (ns/op) | N/A | 11.19 s | ✅ New |
| concurrent_text_delete_range_1000/ (B/op) | N/A | 605.70 MB | ✅ New |
| concurrent_text_delete_range_1000/ (allocs/op) | N/A | 11,544,096 allocs | ✅ New |
| concurrent_tree_delete_range_100/ (ns/op) | N/A | 1.36 s | ✅ New |
| concurrent_tree_delete_range_100/ (B/op) | N/A | 109.04 MB | ✅ New |
| concurrent_tree_delete_range_100/ (allocs/op) | N/A | 1,277,662 allocs | ✅ New |
| concurrent_tree_delete_range_1000/ (ns/op) | N/A | 12.18 s | ✅ New |
| concurrent_tree_delete_range_1000/ (B/op) | N/A | 1.56 GB | ✅ New |
| concurrent_tree_delete_range_1000/ (allocs/op) | N/A | 12,172,630 allocs | ✅ New |
| concurrent_text_edit_delete_all_100/ (ns/op) | N/A | 1.16 s | ✅ New |
| concurrent_text_edit_delete_all_100/ (B/op) | N/A | 83.70 MB | ✅ New |
| concurrent_text_edit_delete_all_100/ (allocs/op) | N/A | 1,022,276 allocs | ✅ New |
| concurrent_text_edit_delete_all_1000/ (ns/op) | N/A | 9.38 s | ✅ New |
| concurrent_text_edit_delete_all_1000/ (B/op) | N/A | 505.43 MB | ✅ New |
| concurrent_text_edit_delete_all_1000/ (allocs/op) | N/A | 9,372,208 allocs | ✅ New |
| concurrent_tree_edit_delete_all_100/ (ns/op) | N/A | 1.18 s | ✅ New |
| concurrent_tree_edit_delete_all_100/ (B/op) | N/A | 94.45 MB | ✅ New |
| concurrent_tree_edit_delete_all_100/ (allocs/op) | N/A | 1,083,656 allocs | ✅ New |
| concurrent_tree_edit_delete_all_1000/ (ns/op) | N/A | 10.19 s | ✅ New |
| concurrent_tree_edit_delete_all_1000/ (B/op) | N/A | 1.27 GB | ✅ New |
| concurrent_tree_edit_delete_all_1000/ (allocs/op) | N/A | 10,024,354 allocs | ✅ New |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1214 +/- ##
=======================================
Coverage 37.41% 37.41%
=======================================
Files 177 177
Lines 27101 27101
=======================================
Hits 10141 10141
Misses 16114 16114
Partials 846 846 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (8)
test/bench/document_bench_test.go (8)
525-528: Consider adding larger tree deletion benchmarksWhile you've included large text benchmarks (100000 characters), the tree benchmarks only go up to 1000 nodes. Consider adding a larger benchmark (e.g., 10000 nodes) for consistency with other benchmarks.
822-846: Consider extracting magic numbers into constantsThe function correctly implements the benchmark for deleting all text at once, but it uses magic numbers for operations. Consider extracting values like deletion range sizes into named constants for better readability.
+// Constants for benchmark configuration +const ( + deleteRangeSize = 10 +) func benchmarkTextDeleteAll(cnt int, b *testing.B) { // existing implementation... }
876-902: Range deletion implementation could be more efficientThe current implementation always deletes from position 0, which means it's deleting from the beginning of the document each time. In real-world scenarios, deletions often happen at different positions. Consider implementing a more varied deletion pattern.
func benchmarkTextDeleteRange(cnt int, b *testing.B) { // ...existing code... err = doc.Update(func(root *json.Object, p *presence.Presence) error { text := root.GetText("k1") for i := 0; i < deleteCount; i++ { - text.Edit(0, deleteRangeSize, "") + // Delete from different positions to simulate real-world usage + startPos := i * (cnt - deleteCount*deleteRangeSize) / deleteCount + text.Edit(startPos, startPos + deleteRangeSize, "") } return nil }) // ...existing code... }
1013-1090: Consider adding comments about tree node indexingThe tree deletion benchmark is well-implemented, but tree node indexing (starting from 1 instead of 0) might be confusing to readers unfamiliar with the implementation. Consider adding a comment explaining this indexing choice.
func benchmarkConcurrentTreeDeleteRange(cnt, clientCount int, b *testing.B) { // ...existing code... + // Note: Tree node indexing starts from 1, not 0 like array indexing for k := 1; k < deleteCount+1; k++ { // ...existing code... } // ...existing code... }
1092-1147: Reduce code duplication between text and tree benchmark functionsThe concurrent delete all benchmarks for text and tree are nearly identical in structure. Consider refactoring to reduce duplication by creating a common benchmark function that takes a type parameter.
+// Common parameters for type-specific operations +type benchmarkTypeOperations struct { + typeName string + initializeFunc func(*json.Object) + editFunc func(*json.Object, int) error + deleteAllFunc func(*json.Object, int) error + validateFunc func(doc1, doc2 *document.Document) bool +} + +func benchmarkConcurrentDeleteAll(cnt, clientCount int, ops benchmarkTypeOperations, b *testing.B) { + // Implementation that works for both text and tree types + // using the operations provided in the ops parameter +}Also applies to: 1149-1208
959-975: Optimize client synchronization for benchmark efficiencyIn multiple benchmark functions, each client synchronizes after every operation, which is realistic but might significantly impact benchmark performance. Consider adding a variant that batches operations before synchronizing to measure the overhead of frequent synchronization.
// Example modification to benchmarkConcurrentTextDeleteAll func benchmarkConcurrentTextDeleteAllBatched(cnt, clientCount int, b *testing.B) { // Similar to existing function, but sync only every N operations batchSize := 10 // ... for k := 0; k < cnt; k++ { clientIdx := k % clientCount - assert.NoError(b, clients[clientIdx].Sync(ctx)) + if k % batchSize == 0 { + assert.NoError(b, clients[clientIdx].Sync(ctx)) + } // ...edit operations... - assert.NoError(b, clients[clientIdx].Sync(ctx)) + if k % batchSize == batchSize-1 { + assert.NoError(b, clients[clientIdx].Sync(ctx)) + } } // Final sync for all clients }Also applies to: 1039-1054, 1113-1129, 1174-1190
979-1002: Verify concurrent operation correctness with strict assertionsThe concurrent benchmarks correctly test that documents converge to the same state. Consider adding additional assertions to verify that the document has the expected content structure or length after concurrent operations to catch subtle CRDT issues.
// After all clients are synced assert.Equal(b, docs[0].Root().GetText("k1").Marshal(), docs[clientCount-1].Root().GetText("k1").Marshal()) +// Verify the expected structure +text := docs[0].Root().GetText("k1") +// Calculate expected length based on operations +expectedLen := cnt - (deleteCount * deleteRangeSize) + deleteCount // Account for "b" insertions +textLen := 0 +err = docs[0].Update(func(root *json.Object, p *presence.Presence) error { + textLen = len(root.GetText("k1").Marshal()) + return nil +}) +assert.True(b, textLen > 0, "Text should not be empty")Also applies to: 1058-1081
502-504: Consider making log level configurableThe benchmark function hardcodes the log level to "error". Consider making this configurable or using a helper function to ensure consistent logging across benchmarks.
-err := logging.SetLogLevel("error") +err := logging.SetLogLevel(getLogLevelForBenchmark()) assert.NoError(b, err)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
test/bench/document_bench_test.go(3 hunks)
🧰 Additional context used
🧬 Code Definitions (1)
test/bench/document_bench_test.go (3)
pkg/document/presence/presence.go (1)
Presence(28-31)pkg/document/key/key.go (1)
Key(32-32)test/helper/helper.go (1)
CleanupClients(579-584)
🔇 Additional comments (6)
test/bench/document_bench_test.go (6)
499-514: Well-structured benchmarking function setupThe new
BenchmarkDocumentDeletionfunction provides a comprehensive suite of benchmark tests for deletion operations. The setup with server initialization and proper teardown using defer is a good practice.
515-523: Good coverage of single-client text deletion scenariosThe benchmarks for single client text deletion with different sizes (10000 and 100000) provide valuable performance insights for simple deletion operations.
530-548: Well-structured range deletion benchmarksThese range deletion benchmarks provide good coverage of incremental deletion operations for both text and tree structures, which complements the "delete all" benchmarks well.
550-568: Good concurrent delete range test implementationThe concurrent deletion range tests effectively simulate real-world collaborative editing scenarios where one client deletes while others insert content.
570-588: Comprehensive concurrent edit-delete all scenariosThese tests effectively benchmark the important scenario where multiple clients add content while one client deletes everything, which is a common collaborative editing pattern.
939-1011: Well-implemented concurrent text deletion benchmarkThis benchmark effectively simulates a realistic concurrent editing scenario with multiple clients. The code properly initializes clients, performs operations, syncs between clients, and verifies the consistency of the final document state.
hackerwins
approved these changes
Apr 4, 2025
hackerwins
left a comment
hackerwins
left a comment
Member
There was a problem hiding this comment.
Thanks for your contribution.
hackerwins
changed the title
3 tasks
3 tasks
Merged
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR adds additional benchmark test cases for concurrent editing and deletion in both text and tree document types.
Previously, only
benchmarkTextDeleteAllandbenchmarkTreeDeleteAlltests existed, which measured performance in a single-client scenario where all content is deleted at once.This update introduces new benchmark cases where multiple clients concurrently perform deletions and insertions. These include:
Which issue(s) this PR fixes:
Related #1208
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
Additional documentation:
Checklist:
Summary by CodeRabbit