[CICD] Benchmark testing that supports training and inference

[wzj09]

PR Category

CICD

PR Types

Test Case

PR Description

Benchmark testing that supports training and inference

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive benchmark testing framework for both training and inference tasks. The changes include new benchmark scripts, test execution scripts, and numerous configuration files for various test cases. My review has identified several critical issues that could lead to script crashes, such as potential division-by-zero errors and the use of undefined variables in test validation code. Additionally, I've found a high-severity issue related to how test cases are parsed from configuration files and an inconsistency in one of the test case setups that would cause failures. I have provided specific code suggestions to address these problems and ensure the new framework is robust and reliable.

[gemini-code-assist]

The script will crash if cfg.generate.num_iters is set to 0. In this scenario, latencies will be an empty list, causing statistics.mean(latencies) on line 64 to raise a statistics.StatisticsError. Furthermore, if total_time is zero (which can happen if all latencies are zero), a ZeroDivisionError will occur on line 68. These edge cases should be handled to prevent the script from crashing.

Suggested change

	avg_latency = statistics.mean(latencies)
	p50 = statistics.median(latencies)
	total_time = sum(latencies)
	throughput = total_tokens / total_time
	if not latencies:
	print("No benchmark iterations were run. Skipping stats.")
	return
	avg_latency = statistics.mean(latencies)
	p50 = statistics.median(latencies)
	total_time = sum(latencies)
	throughput = total_tokens / total_time if total_time > 0 else 0.0

[wzj09]

done

[gemini-code-assist]

The script will crash if cfg.generate.num_iters is set to 0. In this case, latencies will be an empty list, and statistics.mean(latencies) on line 94 will raise a statistics.StatisticsError. It's important to handle this edge case by checking if latencies is empty before computing statistics.

Suggested change

	avg_latency = statistics.mean(latencies)
	p50_latency = statistics.median(latencies)
	if not latencies:
	print("No benchmark iterations were run. Skipping stats.")
	return
	avg_latency = statistics.mean(latencies)
	p50_latency = statistics.median(latencies)

[wzj09]

done

[gemini-code-assist]

The variables perf_threshold and time_threshold used in the f-strings for the assertion messages are not defined within the scope of the test_train_equal function. This will raise a NameError when the tests are executed, causing a crash.

Suggested change

	assert median_tokens_sec >= gold_median_tokens * 0.95, (
	f"Throughput dropped below {perf_threshold*100:.0f}% of gold baseline!"
	)
	assert median_iter_time <= gold_median_time * 1.05, (
	f"Iteration time slower than {time_threshold:.1f}x of gold baseline!"
	)
	perf_threshold = 0.95
	assert median_tokens_sec >= gold_median_tokens * perf_threshold, (
	f"Throughput dropped below {perf_threshold*100:.0f}% of gold baseline!"
	)
	time_threshold = 1.05
	assert median_iter_time <= gold_median_time * time_threshold, (
	f"Iteration time slower than {time_threshold:.1f}x of gold baseline!"
	)

[wzj09]

done

[gemini-code-assist]

The print(result) statement will output the Python list representation (e.g., ['-tp2_pp2', '-tp4_pp2']) to stdout. The calling shell script, test_task.sh, expects a space-separated string to correctly parse the values into a bash array. This discrepancy will lead to parsing errors in the shell script.

Suggested change

	print(result)
	print(*result)

[gemini-code-assist]

The test case robobrain2-flaggems is configured as a functional test rather than a benchmark test, which is inconsistent with the other tests in this benchmark suite.

• Its configuration in tests/benchmarks/test_cases/inference/robobrain2-flaggems/conf/7b-tp2.yaml points to a functional test entrypoint (flagscale/inference/inference_robobrain2.py) and an exp_dir inside tests/functional_tests.
• The gold result file tests/benchmarks/test_cases/inference/robobrain2-flaggems/results_gold/7b-tp2 is not in JSON format and contains text output, which will cause the test_inference_equal pytest function to fail with a JSON decoding error.

While this test is currently commented out, these files introduce inconsistency and will cause failures if the test is re-enabled. Please either correct the configuration to be a proper benchmark test or remove these inconsistent files.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}