-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AMD Slow Memory Bandwidth Allocation(SMBA) #260
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
11 tasks
babumoger
force-pushed
the
AMD-SMBA-0.3
branch
from
January 22, 2024 21:25
fe5c270
to
96db1a4
Compare
rkanagar
requested changes
Feb 9, 2024
babumoger
force-pushed
the
AMD-SMBA-0.3
branch
from
February 16, 2024 20:26
96db1a4
to
32b9f8b
Compare
rkanagar
reviewed
Feb 22, 2024
rkanagar
reviewed
Feb 23, 2024
Adds the new AMD feature X86_FEATURE_SMBA. With this feature, the QOS enforcement policies can be applied to external slow memory connected to the host. QOS enforcement is accomplished by assigning a Class Of Service (COS) to a processor and specifying allocations or limits for that COS for each resource to be allocated. This feature is identified by the CPUID Function 8000_0020_EBX_x0. CPUID Fn8000_0020_EBX_x0 AMD Bandwidth Enforcement Feature Identifiers (ECX=0) Bits Field Name Description 2 L3SBE L3 external slow memory bandwidth enforcement Feature descriptions are available in the specification, "AMD64 Technology Platform Quality of Service Extensions, Revision: 1.03 Publication # 56375 Revision: 1.03 Issue Date: February 2022". Link: https://www.amd.com/en/support/tech-docs/amd64-technology-platform-quality-service-extensions Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537 Signed-off-by: Babu Moger <[email protected]>
This feature is identified by the CPUID Function 8000_0020_EBX_x0. CPUID Fn8000_0020_EBX_x0 AMD Bandwidth Enforcement Feature Identifiers (ECX=0) Bits Field Name Description 2 L3SBE L3 external slow memory bandwidth enforcement Discover the feature and its details using the CPUID Function 8000_0020. Signed-off-by: Babu Moger <[email protected]>
The sysfs interface will have a new directory "/sys/fs/resctrl/info/SMBA/" if the feature SMBA is supported. Read the information from the directory. Signed-off-by: Babu Moger <[email protected]>
SMBA allocation is achieved by writing to the MSRs called L3QOS_SLOWBW_CONTROL_n where "n" is the corresponding COS number. These MSRs begin at C000_0280h. Signed-off-by: Babu Moger <[email protected]>
The pqos_cpu_get_one_by_smba_id retrieves core information from cpu info structure for smba_id. Also initialize the info->smba_id to l3_id. Resource ids are same across all the resources for AMD systems. These functions will be used in API functions smba_get and smba_set. Signed-off-by: Babu Moger <[email protected]>
The function pqos_smba_get_cos_num retrieves the total number of slow memory B/W allocation classes of service from cap structure. These functions will be used in API functions smba_get and smba_set. Signed-off-by: Babu Moger <[email protected]>
Reads the current SMBA allocation for the specific COS using the MSR method. Signed-off-by: Babu Moger <[email protected]>
The function hw_smba_set_amd writes the given SMBA allocation value to the specific class of service using the MSR method. Signed-off-by: Babu Moger <[email protected]>
Add the SMBA related data structure in resctrl_schemata structure. Signed-off-by: Babu Moger <[email protected]>
The function pqos_cpu_get_smba_ids retrieves the smba id's from cpu info structure. These are support functions which will be used to implement os_smba_get and os_smba_set routines. Signed-off-by: Babu Moger <[email protected]>
The function resctrl_schemata_smba_get reads the SMBA class of service information for a given smba_id. These are support functions which will be used to implement os_smba_get and os_smba_set routines. Signed-off-by: Babu Moger <[email protected]>
Update the function resctrl_schemata_alloc to allocate memory for SMBA schemata structure if the SMBA feature is supported. Signed-off-by: Babu Moger <[email protected]>
Update the functions resctrl_schemata_type_get, resctrl_schemata_type_set and related functions to support the SMBA feature. Signed-off-by: Babu Moger <[email protected]>
The function os_smba_get_amd reads all the COS(class of service) information from the sysfs interface. Signed-off-by: Babu Moger <[email protected]>
Add resctrl_schemata_smba_set and resctrl_schemata_write functions to support the SMBA feature. Signed-off-by: Babu Moger <[email protected]>
The function os_smba_set_amd updates the schemata value of a given class of service (COS), using the sysfs interface. Signed-off-by: Babu Moger <[email protected]>
Introduce the function set_smba_cos to apply the class of service allocations. This will be called from alloc_apply if SMBA feature is supported. Signed-off-by: Babu Moger <[email protected]>
Users can now set SMBA feature setting using the pqos tool. E.g. "pqos -e smba:1=64" This means that COS1 on all sockets, can utilize up to 64 units of available slow memory bandwidth. Signed-off-by: Babu Moger <[email protected]>
Introduce cap_print_features_smba to print SMBA capabilities. Signed-off-by: Babu Moger <[email protected]>
Update print_per_socket_config to print the SMBA capabilities for each socket. Signed-off-by: Babu Moger <[email protected]>
Introduce _pqos_cap_smba_change to reset SMBA settings when the user resets the allocations. Signed-off-by: Babu Moger <[email protected]>
Updates the following functions to reset the SMBA schemata to default value. hw_alloc_reset hw_alloc_reset_cos os_alloc_reset os_alloc_reset_schematas os_alloc_reset_light os_alloc_reset_full resctrl_schemata_reset Skip pqos_alloc_reset as this is deprecated. Signed-off-by: Babu Moger <[email protected]>
Update the rdt structure rdt_cfg and rdt_config to include smba fields. Signed-off-by: Babu Moger <[email protected]>
Introduce SMBA feature string in rdtset command interface. E.g. rdtset -t 'smba=128;cpu=8-15' -c 8-15 ls The above command creates a the class of service with cores 8-15 and SMBA settings to 128 and then runs the command "ls". Signed-off-by: Babu Moger <[email protected]>
Update alloc_init to detect SMBA feature and alloc_fini for cleanup. Signed-off-by: Babu Moger <[email protected]>
Updates the function cfg_configure_cos to accept the new feature SMBA. Signed-off-by: Babu Moger <[email protected]>
Update the function cfg_set_pids to accept the new parameter for SMBA feature. Signed-off-by: Babu Moger <[email protected]>
Update the function cfg_set_cores_os to accept the new parameter to support SMBA feature. Signed-off-by: Babu Moger <[email protected]>
Update the function cfg_set_cores_msr to accept the new parameter for SMBA feature. Signed-off-by: Babu Moger <[email protected]>
Signed-off-by: Babu Moger <[email protected]>
Signed-off-by: Babu Moger <[email protected]>
babumoger
force-pushed
the
AMD-SMBA-0.3
branch
from
February 26, 2024 21:51
32b9f8b
to
44a7e02
Compare
This PR is merged into master |
Raghavan, Thank you. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add new AMD feature X86_FEATURE_SMBA. With this feature, the QOS enforcement policies can be applied to external slow memory connected to the host. QOS enforcement is accomplished by assigning a Class Of Service (COS) to a processor and specifying allocations or limits for that COS for each resource to be allocated.
Kernel changes are already committed. Here are the commit ids.
5b6fac3fa44baf x86/resctrl: Detect and configure Slow Memory Bandwidth Allocation
a76f65c89f928 x86/resctrl: Include new features in command line options
78335aac6156 x86/cpufeatures: Add Bandwidth Monitoring Event Configuration feature flag
a5b699665580 x86/resctrl: Add a new resource type RDT_RESOURCE_SMBA
f334f723a63cfc x86/cpufeatures: Add Slow Memory Bandwidth Allocation feature flag
Description
This feature is identified by the CPUID Function 8000_0020_EBX_x0.
CPUID Fn8000_0020_EBX_x0 AMD Bandwidth Enforcement Feature Identifiers (ECX=0)
Bits Field Name Description
2 L3SBE L3 external slow memory bandwidth enforcement
CXL.memory is the only supported "slow" memory device. With the support of SMBA feature, the hardware enables bandwidth allocation on the slow memory devices. If there are multiple slow memory devices in the system, then the throttling logic groups all the slow sources together and applies the limit on them as a whole.
The presence of the SMBA feature(with CXL.memory) is independent of whether a slow memory device is present in the system. If there is no slow memory in the system, then setting an SMBA limit will have no impact on the performance of the system.
The presence of CXL memory can be identified by numactl command.
$numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
node 0 size: 63678 MB node 0 free: 59542 MB
node 1 cpus:
node 1 size: 16122 MB
node 1 free: 15627 MB
node distances:
node 0 1
0: 10 50
1: 50 10
CPU list for CXL memory will be empty. The cpu-cxl node distance is
greater than cpu-to-cpu distances. Node 1 has the CXL memory in this
case. CXL memory can also be identified using ACPI SRAT table and
memory maps.
Feature description is available in the specification, "AMD64
Technology Platform Quality of Service Extensions, Revision: 1.03
Publication # 56375 Revision: 1.03 Issue Date: February 2022".
Link: https://www.amd.com/en/support/tech-docs/amd64-technology-platform-quality-service-extensions
Affected parts
Motivation and Context
With this feature, the QOS enforcement policies can be applied to the external slow memory connected to the host. QOS enforcement is accomplished by assigning a Class Of Service (COS) to a processor and specifying allocations or limits for that COS for each resource to be allocated.
How Has This Been Tested?
Tested on AMD machine and sanity tested on Intel machine to verify for regression
Types of changes
Checklist: