Skip to content

cache keys in iters to reduce gc#9638

Merged
jycor merged 9 commits intomainfrom
james/bytes
Aug 11, 2025
Merged

cache keys in iters to reduce gc#9638
jycor merged 9 commits intomainfrom
james/bytes

Conversation

@jycor
Copy link
Copy Markdown
Contributor

@jycor jycor commented Aug 6, 2025 •
edited
Loading

@jycor jycor marked this pull request as draft August 6, 2025 23:44
@jycor
Copy link
Copy Markdown
Contributor Author

jycor commented Aug 6, 2025

#benchmark

@github-actions
Copy link
Copy Markdown

github-actions bot commented Aug 6, 2025

@jycor workflow run: https://github.com/dolthub/dolt/actions/runs/16791314940

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 7, 2025

@jycor DOLT

test_name from_latency_p95 to_latency_p95 percent_change
tpcc-scale-factor-1 62.19 66.84 7.48
test_name from_server_name from_server_version from_tps to_server_name to_server_version to_tps percent_change
tpcc-scale-factor-1 dolt a02b877 38.56 dolt 240feb5 38.91 0.91

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 7, 2025

@jycor DOLT

comparing_percentages
100.000000 to 100.000000
version result total
240feb5 ok 5937457
version total_tests
240feb5 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 7, 2025

@jycor DOLT

read_tests from_latency to_latency percent_change
covering_index_scan 0.67 0.68 1.49
groupby_scan 19.65 19.65 0.0
index_join 2.52 2.52 0.0
index_join_scan 1.44 1.47 2.08
index_scan 30.81 31.94 3.67
oltp_point_select 0.28 0.29 3.57
oltp_read_only 5.28 5.37 1.7
select_random_points 0.6 0.61 1.67
select_random_ranges 0.62 0.64 3.23
table_scan 32.53 33.12 1.81
types_table_scan 125.52 127.81 1.82
write_tests from_latency to_latency percent_change
oltp_delete_insert 6.55 6.55 0.0
oltp_insert 3.25 3.25 0.0
oltp_read_write 11.65 11.87 1.89
oltp_update_index 3.3 3.3 0.0
oltp_update_non_index 3.25 3.25 0.0
oltp_write_only 6.43 6.55 1.87
types_delete_insert 6.91 6.91 0.0

@jycor
Copy link
Copy Markdown
Contributor Author

jycor commented Aug 7, 2025

#benchmark

@github-actions
Copy link
Copy Markdown

github-actions bot commented Aug 7, 2025

@jycor workflow run: https://github.com/dolthub/dolt/actions/runs/16811446923

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 7, 2025

@jycor DOLT

comparing_percentages
100.000000 to 100.000000
version result total
910c8d1 ok 5937457
version total_tests
910c8d1 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 7, 2025

@jycor DOLT

test_name from_latency_p95 to_latency_p95 percent_change
tpcc-scale-factor-1 64.47 66.84 3.68
test_name from_server_name from_server_version from_tps to_server_name to_server_version to_tps percent_change
tpcc-scale-factor-1 dolt 86ec7e6 38.36 dolt 910c8d1 38.76 1.04

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 7, 2025

@jycor DOLT

read_tests from_latency to_latency percent_change
covering_index_scan 0.68 0.67 -1.47
groupby_scan 20.0 19.29 -3.55
index_join 2.52 2.48 -1.59
index_join_scan 1.47 1.42 -3.4
index_scan 31.37 30.81 -1.79
oltp_point_select 0.29 0.28 -3.45
oltp_read_only 5.37 5.28 -1.68
select_random_points 0.61 0.6 -1.64
select_random_ranges 0.64 0.63 -1.56
table_scan 32.53 32.53 0.0
types_table_scan 127.81 127.81 0.0
write_tests from_latency to_latency percent_change
oltp_delete_insert 6.55 6.55 0.0
oltp_insert 3.25 3.25 0.0
oltp_read_write 11.87 11.65 -1.85
oltp_update_index 3.3 3.3 0.0
oltp_update_non_index 3.25 3.25 0.0
oltp_write_only 6.55 6.43 -1.83
types_delete_insert 6.91 6.91 0.0

@jycor
Copy link
Copy Markdown
Contributor Author

jycor commented Aug 7, 2025

#benchmark

@github-actions
Copy link
Copy Markdown

github-actions bot commented Aug 7, 2025

@jycor workflow run: https://github.com/dolthub/dolt/actions/runs/16815856208

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 7, 2025

@jycor DOLT

test_name from_latency_p95 to_latency_p95 percent_change
tpcc-scale-factor-1 64.47 66.84 3.68
test_name from_server_name from_server_version from_tps to_server_name to_server_version to_tps percent_change
tpcc-scale-factor-1 dolt 8e36c39 38.2 dolt 2adec5d 37.69 -1.34

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 7, 2025

@jycor DOLT

read_tests from_latency to_latency percent_change
covering_index_scan 0.67 0.68 1.49
groupby_scan 19.65 19.29 -1.83
index_join 2.52 2.52 0.0
index_join_scan 1.44 1.47 2.08
index_scan 30.26 30.81 1.82
oltp_point_select 0.28 0.28 0.0
oltp_read_only 5.28 5.37 1.7
select_random_points 0.6 0.61 1.67
select_random_ranges 0.62 0.64 3.23
table_scan 31.94 33.12 3.69
types_table_scan 125.52 130.13 3.67
write_tests from_latency to_latency percent_change
oltp_delete_insert 6.55 6.55 0.0
oltp_insert 3.25 3.25 0.0
oltp_read_write 11.65 11.87 1.89
oltp_update_index 3.3 3.3 0.0
oltp_update_non_index 3.25 3.25 0.0
oltp_write_only 6.43 6.55 1.87
types_delete_insert 6.91 6.91 0.0

@jycor
Copy link
Copy Markdown
Contributor Author

jycor commented Aug 7, 2025

#benchmark

@github-actions
Copy link
Copy Markdown

github-actions bot commented Aug 7, 2025

@jycor workflow run: https://github.com/dolthub/dolt/actions/runs/16818167742

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 8, 2025

@jycor DOLT

comparing_percentages
100.000000 to 100.000000
version result total
2e79f9c ok 5937457
version total_tests
2e79f9c 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 8, 2025

@jycor DOLT

test_name from_latency_p95 to_latency_p95 percent_change
tpcc-scale-factor-1 63.32 63.32 0.0
test_name from_server_name from_server_version from_tps to_server_name to_server_version to_tps percent_change
tpcc-scale-factor-1 dolt 8e36c39 38.71 dolt 2e79f9c 39.53 2.12

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 8, 2025

@jycor DOLT

read_tests from_latency to_latency percent_change
covering_index_scan 0.68 0.68 0.0
groupby_scan 20.0 19.29 -3.55
index_join 2.52 2.52 0.0
index_join_scan 1.47 1.47 0.0
index_scan 31.37 31.37 0.0
oltp_point_select 0.29 0.29 0.0
oltp_read_only 5.37 5.37 0.0
select_random_points 0.61 0.61 0.0
select_random_ranges 0.64 0.64 0.0
table_scan 32.53 32.53 0.0
types_table_scan 127.81 130.13 1.82
write_tests from_latency to_latency percent_change
oltp_delete_insert 6.55 6.55 0.0
oltp_insert 3.25 3.25 0.0
oltp_read_write 11.87 11.87 0.0
oltp_update_index 3.3 3.3 0.0
oltp_update_non_index 3.25 3.25 0.0
oltp_write_only 6.55 6.55 0.0
types_delete_insert 6.91 6.91 0.0

@jycor
Copy link
Copy Markdown
Contributor Author

jycor commented Aug 8, 2025

#benchmark

@github-actions
Copy link
Copy Markdown

github-actions bot commented Aug 8, 2025

@jycor workflow run: https://github.com/dolthub/dolt/actions/runs/16837819616

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 8, 2025

@jycor DOLT

comparing_percentages
100.000000 to 100.000000
version result total
809db60 ok 5937457
version total_tests
809db60 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 8, 2025

@jycor DOLT

test_name from_latency_p95 to_latency_p95 percent_change
tpcc-scale-factor-1 61.08 65.65 7.48
test_name from_server_name from_server_version from_tps to_server_name to_server_version to_tps percent_change
tpcc-scale-factor-1 dolt dfb2b4e 38.63 dolt 809db60 38.8 0.44

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 8, 2025

@jycor DOLT

read_tests from_latency to_latency percent_change
covering_index_scan 0.68 0.68 0.0
groupby_scan 20.0 19.29 -3.55
index_join 2.52 2.52 0.0
index_join_scan 1.47 1.44 -2.04
index_scan 31.37 31.37 0.0
oltp_point_select 0.29 0.28 -3.45
oltp_read_only 5.37 5.37 0.0
select_random_points 0.61 0.61 0.0
select_random_ranges 0.64 0.64 0.0
table_scan 32.53 33.12 1.81
types_table_scan 127.81 130.13 1.82
write_tests from_latency to_latency percent_change
oltp_delete_insert 6.67 6.55 -1.8
oltp_insert 3.25 3.25 0.0
oltp_read_write 11.87 11.87 0.0
oltp_update_index 3.3 3.3 0.0
oltp_update_non_index 3.25 3.25 0.0
oltp_write_only 6.55 6.55 0.0
types_delete_insert 6.91 6.91 0.0

@jycor jycor changed the title test cache keys in iters to reduce gc Aug 8, 2025
@jycor jycor marked this pull request as ready for review August 8, 2025 21:22
fulghum
fulghum approved these changes Aug 8, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@fulghum fulghum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

go/libraries/doltcore/sqle/writer/prolly_index_writer.go
valBld *val.TupleBuilder
valMap val.OrdinalMapping

key sql.Row
Copy link
Copy Markdown
Contributor

@fulghum fulghum Aug 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth adding a doc string to explain that this is a buffer that gets reused, as well as for the new field in prollySecondaryIndexWriter.

go/libraries/doltcore/sqle/enginetest/dolt_engine_test.go Outdated
// Convenience test for debugging a single query. Unskip and set to the desired query.
func TestSingleScript(t *testing.T) {
t.Skip()
//t.Skip()
Copy link
Copy Markdown
Contributor

@fulghum fulghum Aug 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean to check this in?

@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 11, 2025

@jycor DOLT

comparing_percentages
100.000000 to 100.000000
version result total
381c70d ok 5937457
version total_tests
381c70d 5937457
correctness_percentage
100.0

@jycor jycor mentioned this pull request Aug 11, 2025
Closed
@coffeegoddd
Copy link
Copy Markdown
Contributor

coffeegoddd commented Aug 11, 2025

@jycor DOLT

comparing_percentages
100.000000 to 100.000000
version result total
504f674 ok 5937457
version total_tests
504f674 5937457
correctness_percentage
100.0

@jycor jycor merged commit 2c9b9a9 into main Aug 11, 2025
20 of 21 checks passed
@jycor jycor deleted the james/bytes branch August 11, 2025 18:24
@BrewTestBot BrewTestBot mentioned this pull request Aug 12, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fulghum fulghum fulghum approved these changes

Assignees

No one assigned

Labels

correctness_approved

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jycor @coffeegoddd @fulghum