Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix benchmarks commands #274

Merged
merged 2 commits into from
Nov 9, 2024
Merged

Fix benchmarks commands #274

merged 2 commits into from
Nov 9, 2024

Conversation

iamazeem
Copy link
Collaborator

@iamazeem iamazeem commented Nov 9, 2024

  • Fix command for tsv-select
    • The output was different for tsv-select.
    • Fixing the command fixed the output.
    • Now, the output is same for zsv, xsv, and tsv-select.
  • For count tests, the equivalent command from tsv-utils seems to be tsv-summarize --count instead of number-lines.
    • Now, the output is similar for all three tools except that counting of header row.

Here's the output of the equivalent count commands:

$ zsv count < worldcitiespop_mil.csv 
1000000

$ xsv count --no-headers < worldcitiespop_mil.csv 
1000001

$ tsv-summarize --count -d, < worldcitiespop_mil.csv 
1000001

Currently, zsv count doesn't support a --no-headers flag hence the difference of 1.

With above changes, ran benchmarks locally using builds from this workflow run:
https://github.com/liquidaty/zsv/actions/runs/11752021827

Here are the results:

gcc

$ ZSV_LINUX_BUILD_COMPILER=gcc ./scripts/ci-run-benchmarks.sh
[INF] Running ./scripts/ci-run-benchmarks.sh
[INF] OS: linux
[INF] RUNS: 6
[INF] SKIP_FIRST_RUN: true
[INF] BENCHMARKS_DIR: .benchmarks
[INF] ZSV_TAG: 0.3.9-alpha
[INF] ZSV_LINUX_BUILD_COMPILER: gcc
[INF] Downloading CSV file... [worldcitiespop_mil.csv] [SKIPPED]
[INF] Downloading... [zsv-0.3.9-alpha-amd64-linux-gcc.tar.gz] [SKIPPED]
[INF] Extracting... [zsv-0.3.9-alpha-amd64-linux-gcc.tar.gz] [DONE]
[INF] Downloading... [tsv-utils-v2.2.0_linux-x86_64_ldc2.tar.gz] [SKIPPED]
[INF] Extracting... [tsv-utils-v2.2.0_linux-x86_64_ldc2.tar.gz] [DONE]
[INF] Downloading... [xsv-0.13.0-x86_64-unknown-linux-musl.tar.gz] [SKIPPED]
[INF] Extracting... [xsv-0.13.0-x86_64-unknown-linux-musl.tar.gz] [DONE]
[INF] Running count benchmarks...
1 | zsv : real 0.04 user 0.03 sys 0.00
2 | zsv : real 0.03 user 0.03 sys 0.00
3 | zsv : real 0.03 user 0.02 sys 0.00
4 | zsv : real 0.03 user 0.03 sys 0.00
5 | zsv : real 0.03 user 0.03 sys 0.00
1 | xsv : real 0.08 user 0.07 sys 0.00
2 | xsv : real 0.08 user 0.07 sys 0.00
3 | xsv : real 0.08 user 0.07 sys 0.01
4 | xsv : real 0.08 user 0.06 sys 0.01
5 | xsv : real 0.08 user 0.07 sys 0.00
1 | tsv : real 0.02 user 0.01 sys 0.00
2 | tsv : real 0.02 user 0.01 sys 0.00
3 | tsv : real 0.02 user 0.02 sys 0.00
4 | tsv : real 0.02 user 0.01 sys 0.00
5 | tsv : real 0.02 user 0.01 sys 0.00
[INF] Running select benchmarks...
1 | zsv : real 0.09 user 0.08 sys 0.00
2 | zsv : real 0.09 user 0.08 sys 0.00
3 | zsv : real 0.09 user 0.09 sys 0.00
4 | zsv : real 0.09 user 0.09 sys 0.00
5 | zsv : real 0.09 user 0.09 sys 0.00
1 | xsv : real 0.36 user 0.36 sys 0.00
2 | xsv : real 0.36 user 0.35 sys 0.00
3 | xsv : real 0.36 user 0.35 sys 0.00
4 | xsv : real 0.36 user 0.35 sys 0.00
5 | xsv : real 0.36 user 0.36 sys 0.00
1 | tsv : real 0.11 user 0.10 sys 0.00
2 | tsv : real 0.11 user 0.10 sys 0.00
3 | tsv : real 0.11 user 0.09 sys 0.01
4 | tsv : real 0.11 user 0.10 sys 0.01
5 | tsv : real 0.11 user 0.10 sys 0.00
[INF] Generating Markdown output... [benchmarks.md]
[INF] Generated Markdown output successfully!
[INF] --- [DONE] ---

clang

$ ZSV_LINUX_BUILD_COMPILER=clang ./scripts/ci-run-benchmarks.sh
[INF] Running ./scripts/ci-run-benchmarks.sh
[INF] OS: linux
[INF] RUNS: 6
[INF] SKIP_FIRST_RUN: true
[INF] BENCHMARKS_DIR: .benchmarks
[INF] ZSV_TAG: 0.3.9-alpha
[INF] ZSV_LINUX_BUILD_COMPILER: clang
[INF] Downloading CSV file... [worldcitiespop_mil.csv] [SKIPPED]
[INF] Downloading... [zsv-0.3.9-alpha-amd64-linux-clang.tar.gz] [SKIPPED]
[INF] Extracting... [zsv-0.3.9-alpha-amd64-linux-clang.tar.gz] [DONE]
[INF] Downloading... [tsv-utils-v2.2.0_linux-x86_64_ldc2.tar.gz] [SKIPPED]
[INF] Extracting... [tsv-utils-v2.2.0_linux-x86_64_ldc2.tar.gz] [DONE]
[INF] Downloading... [xsv-0.13.0-x86_64-unknown-linux-musl.tar.gz] [SKIPPED]
[INF] Extracting... [xsv-0.13.0-x86_64-unknown-linux-musl.tar.gz] [DONE]
[INF] Running count benchmarks...
1 | zsv : real 0.05 user 0.04 sys 0.00
2 | zsv : real 0.04 user 0.03 sys 0.00
3 | zsv : real 0.04 user 0.03 sys 0.00
4 | zsv : real 0.04 user 0.03 sys 0.00
5 | zsv : real 0.04 user 0.04 sys 0.00
1 | xsv : real 0.08 user 0.07 sys 0.00
2 | xsv : real 0.08 user 0.07 sys 0.01
3 | xsv : real 0.08 user 0.07 sys 0.00
4 | xsv : real 0.08 user 0.08 sys 0.00
5 | xsv : real 0.08 user 0.07 sys 0.00
1 | tsv : real 0.02 user 0.02 sys 0.00
2 | tsv : real 0.02 user 0.01 sys 0.01
3 | tsv : real 0.02 user 0.02 sys 0.00
4 | tsv : real 0.02 user 0.02 sys 0.00
5 | tsv : real 0.02 user 0.01 sys 0.00
[INF] Running select benchmarks...
1 | zsv : real 0.09 user 0.08 sys 0.00
2 | zsv : real 0.10 user 0.09 sys 0.00
3 | zsv : real 0.09 user 0.09 sys 0.00
4 | zsv : real 0.10 user 0.09 sys 0.00
5 | zsv : real 0.09 user 0.08 sys 0.00
1 | xsv : real 0.36 user 0.35 sys 0.01
2 | xsv : real 0.37 user 0.36 sys 0.00
3 | xsv : real 0.36 user 0.34 sys 0.01
4 | xsv : real 0.36 user 0.35 sys 0.00
5 | xsv : real 0.36 user 0.35 sys 0.00
1 | tsv : real 0.11 user 0.10 sys 0.00
2 | tsv : real 0.11 user 0.11 sys 0.00
3 | tsv : real 0.10 user 0.10 sys 0.00
4 | tsv : real 0.11 user 0.10 sys 0.00
5 | tsv : real 0.11 user 0.11 sys 0.00
[INF] Generating Markdown output... [benchmarks.md]
[INF] Generated Markdown output successfully!
[INF] --- [DONE] ---

musl

$ ZSV_LINUX_BUILD_COMPILER=musl ./scripts/ci-run-benchmarks.sh
[INF] Running ./scripts/ci-run-benchmarks.sh
[INF] OS: linux
[INF] RUNS: 6
[INF] SKIP_FIRST_RUN: true
[INF] BENCHMARKS_DIR: .benchmarks
[INF] ZSV_TAG: 0.3.9-alpha
[INF] ZSV_LINUX_BUILD_COMPILER: musl
[INF] Downloading CSV file... [worldcitiespop_mil.csv] [SKIPPED]
[INF] Downloading... [zsv-0.3.9-alpha-amd64-linux-musl.tar.gz] [SKIPPED]
[INF] Extracting... [zsv-0.3.9-alpha-amd64-linux-musl.tar.gz] [DONE]
[INF] Downloading... [tsv-utils-v2.2.0_linux-x86_64_ldc2.tar.gz] [SKIPPED]
[INF] Extracting... [tsv-utils-v2.2.0_linux-x86_64_ldc2.tar.gz] [DONE]
[INF] Downloading... [xsv-0.13.0-x86_64-unknown-linux-musl.tar.gz] [SKIPPED]
[INF] Extracting... [xsv-0.13.0-x86_64-unknown-linux-musl.tar.gz] [DONE]
[INF] Running count benchmarks...
1 | zsv : real 0.03 user 0.03 sys 0.00
2 | zsv : real 0.03 user 0.03 sys 0.00
3 | zsv : real 0.03 user 0.03 sys 0.00
4 | zsv : real 0.03 user 0.03 sys 0.00
5 | zsv : real 0.03 user 0.02 sys 0.00
1 | xsv : real 0.08 user 0.07 sys 0.00
2 | xsv : real 0.08 user 0.08 sys 0.00
3 | xsv : real 0.08 user 0.07 sys 0.00
4 | xsv : real 0.08 user 0.07 sys 0.00
5 | xsv : real 0.08 user 0.08 sys 0.00
1 | tsv : real 0.02 user 0.01 sys 0.01
2 | tsv : real 0.02 user 0.01 sys 0.00
3 | tsv : real 0.02 user 0.01 sys 0.00
4 | tsv : real 0.02 user 0.02 sys 0.00
5 | tsv : real 0.02 user 0.02 sys 0.00
[INF] Running select benchmarks...
1 | zsv : real 0.25 user 0.24 sys 0.00
2 | zsv : real 0.26 user 0.25 sys 0.00
3 | zsv : real 0.26 user 0.26 sys 0.00
4 | zsv : real 0.25 user 0.24 sys 0.00
5 | zsv : real 0.25 user 0.25 sys 0.00
1 | xsv : real 0.36 user 0.35 sys 0.01
2 | xsv : real 0.36 user 0.35 sys 0.01
3 | xsv : real 0.36 user 0.36 sys 0.00
4 | xsv : real 0.36 user 0.36 sys 0.00
5 | xsv : real 0.36 user 0.34 sys 0.01
1 | tsv : real 0.11 user 0.10 sys 0.00
2 | tsv : real 0.11 user 0.10 sys 0.00
3 | tsv : real 0.11 user 0.11 sys 0.00
4 | tsv : real 0.10 user 0.10 sys 0.00
5 | tsv : real 0.11 user 0.10 sys 0.00
[INF] Generating Markdown output... [benchmarks.md]
[INF] Generated Markdown output successfully!
[INF] --- [DONE] ---

Signed-off-by: Azeem Sajid [email protected]

@iamazeem iamazeem self-assigned this Nov 9, 2024
@iamazeem iamazeem added the enhancement New feature or request label Nov 9, 2024
@iamazeem iamazeem requested a review from liquidaty November 9, 2024 10:55
@iamazeem iamazeem mentioned this pull request Nov 9, 2024
Draft
@iamazeem
Copy link
Collaborator Author

iamazeem commented Nov 9, 2024

See the latest musl build benchmarks in #263 (comment).

@iamazeem iamazeem merged commit 225d52e into main Nov 9, 2024
1 of 4 checks passed
@iamazeem iamazeem deleted the fix-benchmarks branch November 9, 2024 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liquidaty liquidaty liquidaty approved these changes

Assignees

@iamazeem iamazeem

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@iamazeem @liquidaty