Repository navigation
Conversation
rdc_job_get_stats() returned from inside its per-GPU loop, so only the GPU with the lowest ID was summarized, and the entries of the other GPUs were not filled in. It also wrote each GPU's entry at gpus[gpu_id], past the 16-entry array for a GPU ID above 15 or a partition entity, which crashed rdcd. Fill gpus[0] to gpus[num_gpus - 1], ordered by gpu_id, after zeroing the output and marking every statistic as without samples. Average each summary statistic after the loop over the GPUs that have samples for it; the PCIe total had no averaging step. rdc_job_start_stats() rejects a group of more than RDC_MAX_NUM_GPUS_PER_JOB GPUs with RDC_ST_MAX_LIMIT, and rdc_job_get_stats() checks the same bound before it fills anything. A GPU without a total memory reading, such as a partition, reports memory utilization as unavailable instead of failing the call. The divisor for memory utilization, 1% of the total memory, was truncated to 32 bits, which inflated it on GPUs with 400 GiB of memory or more. rdci stats -j labels each GPU by its ID instead of its position. A statistic without samples prints as N/A, and as null values in JSON, instead of zeros and a minimum of 18446744073709551615. entity_to_string() moves to RdciSubSystem.cc, shared by the dmon and stats subsystems. test_fake_gpu_job_stats.py runs jobs on GPUs 1 and 3, on GPUs 51 to 66, on a partition alone and with a GPU, and on 17 GPUs against the fake amd-smi, and reads each job while it runs and after it stops. Signed-off-by: Adam360x <Adam.pryor@amd.com>
Contributor
Author
|
Reviewed, this one looks good |
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
rdc_job_get_stats()returned from inside its per-GPU loop, so only the GPU with the lowest ID was summarized and the other GPUs' entries were not filled in. It wrote each GPU's entry atgpus[gpu_id], past the 16-entry array for a GPU ID above 15 or a partition entity, which crashedrdcd.Technical Details
gpus[0]togpus[num_gpus - 1], ordered by GPU ID, after zeroing the output and marking every statistic as without samples. Average each summary statistic over the GPUs that have samples for it (the PCIe total had no averaging step).rdc_job_start_stats()rejects a group of more thanRDC_MAX_NUM_GPUS_PER_JOBGPUs withRDC_ST_MAX_LIMIT, andrdc_job_get_stats()checks the same bound before it fills anything.rdci stats -jlabels each GPU by its ID. A statistic without samples prints as N/A (null in JSON) instead of zeros and a minimum of 18446744073709551615.Issue Tracking
ISSUE ID : #14107
Part of a series of RDC fixes, which #14107 lists. No JIRA ticket.
Test Plan
test_fake_gpu_job_stats.pyruns jobs against the fake amd-smi on GPUs 1 and 3, on GPUs 51 to 66, on a partition alone and with a GPU, and on 17 GPUs, and reads each job while it runs and after it stops.Test Result
Fake-GPU suite: 110 passed, 24 skipped.
tests/ci: 92 passed, 6 skipped.Submission Checklist