Skip to content

Commit 75ef718

Browse files
gormanmtorvalds
gormanm
authored and
torvalds
committed
mm, vmstat: add infrastructure for per-node vmstats
Patchset: "Move LRU page reclaim from zones to nodes v9" This series moves LRUs from the zones to the node. While this is a current rebase, the test results were based on mmotm as of June 23rd. Conceptually, this series is simple but there are a lot of details. Some of the broad motivations for this are; 1. The residency of a page partially depends on what zone the page was allocated from. This is partially combatted by the fair zone allocation policy but that is a partial solution that introduces overhead in the page allocator paths. 2. Currently, reclaim on node 0 behaves slightly different to node 1. For example, direct reclaim scans in zonelist order and reclaims even if the zone is over the high watermark regardless of the age of pages in that LRU. Kswapd on the other hand starts reclaim on the highest unbalanced zone. A difference in distribution of file/anon pages due to when they were allocated results can result in a difference in again. While the fair zone allocation policy mitigates some of the problems here, the page reclaim results on a multi-zone node will always be different to a single-zone node. it was scheduled on as a result. 3. kswapd and the page allocator scan zones in the opposite order to avoid interfering with each other but it's sensitive to timing. This mitigates the page allocator using pages that were allocated very recently in the ideal case but it's sensitive to timing. When kswapd is allocating from lower zones then it's great but during the rebalancing of the highest zone, the page allocator and kswapd interfere with each other. It's worse if the highest zone is small and difficult to balance. 4. slab shrinkers are node-based which makes it harder to identify the exact relationship between slab reclaim and LRU reclaim. The reason we have zone-based reclaim is that we used to have large highmem zones in common configurations and it was necessary to quickly find ZONE_NORMAL pages for reclaim. Today, this is much less of a concern as machines with lots of memory will (or should) use 64-bit kernels. Combinations of 32-bit hardware and 64-bit hardware are rare. Machines that do use highmem should have relatively low highmem:lowmem ratios than we worried about in the past. Conceptually, moving to node LRUs should be easier to understand. The page allocator plays fewer tricks to game reclaim and reclaim behaves similarly on all nodes. The series has been tested on a 16 core UMA machine and a 2-socket 48 core NUMA machine. The UMA results are presented in most cases as the NUMA machine behaved similarly. pagealloc --------- This is a microbenchmark that shows the benefit of removing the fair zone allocation policy. It was tested uip to order-4 but only orders 0 and 1 are shown as the other orders were comparable. 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v9 Min total-odr0-1 490.00 ( 0.00%) 457.00 ( 6.73%) Min total-odr0-2 347.00 ( 0.00%) 329.00 ( 5.19%) Min total-odr0-4 288.00 ( 0.00%) 273.00 ( 5.21%) Min total-odr0-8 251.00 ( 0.00%) 239.00 ( 4.78%) Min total-odr0-16 234.00 ( 0.00%) 222.00 ( 5.13%) Min total-odr0-32 223.00 ( 0.00%) 211.00 ( 5.38%) Min total-odr0-64 217.00 ( 0.00%) 208.00 ( 4.15%) Min total-odr0-128 214.00 ( 0.00%) 204.00 ( 4.67%) Min total-odr0-256 250.00 ( 0.00%) 230.00 ( 8.00%) Min total-odr0-512 271.00 ( 0.00%) 269.00 ( 0.74%) Min total-odr0-1024 291.00 ( 0.00%) 282.00 ( 3.09%) Min total-odr0-2048 303.00 ( 0.00%) 296.00 ( 2.31%) Min total-odr0-4096 311.00 ( 0.00%) 309.00 ( 0.64%) Min total-odr0-8192 316.00 ( 0.00%) 314.00 ( 0.63%) Min total-odr0-16384 317.00 ( 0.00%) 315.00 ( 0.63%) Min total-odr1-1 742.00 ( 0.00%) 712.00 ( 4.04%) Min total-odr1-2 562.00 ( 0.00%) 530.00 ( 5.69%) Min total-odr1-4 457.00 ( 0.00%) 433.00 ( 5.25%) Min total-odr1-8 411.00 ( 0.00%) 381.00 ( 7.30%) Min total-odr1-16 381.00 ( 0.00%) 356.00 ( 6.56%) Min total-odr1-32 372.00 ( 0.00%) 346.00 ( 6.99%) Min total-odr1-64 372.00 ( 0.00%) 343.00 ( 7.80%) Min total-odr1-128 375.00 ( 0.00%) 351.00 ( 6.40%) Min total-odr1-256 379.00 ( 0.00%) 351.00 ( 7.39%) Min total-odr1-512 385.00 ( 0.00%) 355.00 ( 7.79%) Min total-odr1-1024 386.00 ( 0.00%) 358.00 ( 7.25%) Min total-odr1-2048 390.00 ( 0.00%) 362.00 ( 7.18%) Min total-odr1-4096 390.00 ( 0.00%) 362.00 ( 7.18%) Min total-odr1-8192 388.00 ( 0.00%) 363.00 ( 6.44%) This shows a steady improvement throughout. The primary benefit is from reduced system CPU usage which is obvious from the overall times; 4.7.0-rc4 4.7.0-rc4 mmotm-20160623nodelru-v8 User 189.19 191.80 System 2604.45 2533.56 Elapsed 2855.30 2786.39 The vmstats also showed that the fair zone allocation policy was definitely removed as can be seen here; 4.7.0-rc3 4.7.0-rc3 mmotm-20160623 nodelru-v8 DMA32 allocs 28794729769 0 Normal allocs 48432501431 77227309877 Movable allocs 0 0 tiobench on ext4 ---------------- tiobench is a benchmark that artifically benefits if old pages remain resident while new pages get reclaimed. The fair zone allocation policy mitigates this problem so pages age fairly. While the benchmark has problems, it is important that tiobench performance remains constant as it implies that page aging problems that the fair zone allocation policy fixes are not re-introduced. 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v9 Min PotentialReadSpeed 89.65 ( 0.00%) 90.21 ( 0.62%) Min SeqRead-MB/sec-1 82.68 ( 0.00%) 82.01 ( -0.81%) Min SeqRead-MB/sec-2 72.76 ( 0.00%) 72.07 ( -0.95%) Min SeqRead-MB/sec-4 75.13 ( 0.00%) 74.92 ( -0.28%) Min SeqRead-MB/sec-8 64.91 ( 0.00%) 65.19 ( 0.43%) Min SeqRead-MB/sec-16 62.24 ( 0.00%) 62.22 ( -0.03%) Min RandRead-MB/sec-1 0.88 ( 0.00%) 0.88 ( 0.00%) Min RandRead-MB/sec-2 0.95 ( 0.00%) 0.92 ( -3.16%) Min RandRead-MB/sec-4 1.43 ( 0.00%) 1.34 ( -6.29%) Min RandRead-MB/sec-8 1.61 ( 0.00%) 1.60 ( -0.62%) Min RandRead-MB/sec-16 1.80 ( 0.00%) 1.90 ( 5.56%) Min SeqWrite-MB/sec-1 76.41 ( 0.00%) 76.85 ( 0.58%) Min SeqWrite-MB/sec-2 74.11 ( 0.00%) 73.54 ( -0.77%) Min SeqWrite-MB/sec-4 80.05 ( 0.00%) 80.13 ( 0.10%) Min SeqWrite-MB/sec-8 72.88 ( 0.00%) 73.20 ( 0.44%) Min SeqWrite-MB/sec-16 75.91 ( 0.00%) 76.44 ( 0.70%) Min RandWrite-MB/sec-1 1.18 ( 0.00%) 1.14 ( -3.39%) Min RandWrite-MB/sec-2 1.02 ( 0.00%) 1.03 ( 0.98%) Min RandWrite-MB/sec-4 1.05 ( 0.00%) 0.98 ( -6.67%) Min RandWrite-MB/sec-8 0.89 ( 0.00%) 0.92 ( 3.37%) Min RandWrite-MB/sec-16 0.92 ( 0.00%) 0.93 ( 1.09%) 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 approx-v9 User 645.72 525.90 System 403.85 331.75 Elapsed 6795.36 6783.67 This shows that the series has little or not impact on tiobench which is desirable and a reduction in system CPU usage. It indicates that the fair zone allocation policy was removed in a manner that didn't reintroduce one class of page aging bug. There were only minor differences in overall reclaim activity 4.7.0-rc4 4.7.0-rc4 mmotm-20160623nodelru-v8 Minor Faults 645838 647465 Major Faults 573 640 Swap Ins 0 0 Swap Outs 0 0 DMA allocs 0 0 DMA32 allocs 46041453 44190646 Normal allocs 78053072 79887245 Movable allocs 0 0 Allocation stalls 24 67 Stall zone DMA 0 0 Stall zone DMA32 0 0 Stall zone Normal 0 2 Stall zone HighMem 0 0 Stall zone Movable 0 65 Direct pages scanned 10969 30609 Kswapd pages scanned 93375144 93492094 Kswapd pages reclaimed 93372243 93489370 Direct pages reclaimed 10969 30609 Kswapd efficiency 99% 99% Kswapd velocity 13741.015 13781.934 Direct efficiency 100% 100% Direct velocity 1.614 4.512 Percentage direct scans 0% 0% kswapd activity was roughly comparable. There were differences in direct reclaim activity but negligible in the context of the overall workload (velocity of 4 pages per second with the patches applied, 1.6 pages per second in the baseline kernel). pgbench read-only large configuration on ext4 --------------------------------------------- pgbench is a database benchmark that can be sensitive to page reclaim decisions. This also checks if removing the fair zone allocation policy is safe pgbench Transactions 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v8 Hmean 1 188.26 ( 0.00%) 189.78 ( 0.81%) Hmean 5 330.66 ( 0.00%) 328.69 ( -0.59%) Hmean 12 370.32 ( 0.00%) 380.72 ( 2.81%) Hmean 21 368.89 ( 0.00%) 369.00 ( 0.03%) Hmean 30 382.14 ( 0.00%) 360.89 ( -5.56%) Hmean 32 428.87 ( 0.00%) 432.96 ( 0.95%) Negligible differences again. As with tiobench, overall reclaim activity was comparable. bonnie++ on ext4 ---------------- No interesting performance difference, negligible differences on reclaim stats. paralleldd on ext4 ------------------ This workload uses varying numbers of dd instances to read large amounts of data from disk. 4.7.0-rc3 4.7.0-rc3 mmotm-20160623 nodelru-v9 Amean Elapsd-1 186.04 ( 0.00%) 189.41 ( -1.82%) Amean Elapsd-3 192.27 ( 0.00%) 191.38 ( 0.46%) Amean Elapsd-5 185.21 ( 0.00%) 182.75 ( 1.33%) Amean Elapsd-7 183.71 ( 0.00%) 182.11 ( 0.87%) Amean Elapsd-12 180.96 ( 0.00%) 181.58 ( -0.35%) Amean Elapsd-16 181.36 ( 0.00%) 183.72 ( -1.30%) 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v9 User 1548.01 1552.44 System 8609.71 8515.08 Elapsed 3587.10 3594.54 There is little or no change in performance but some drop in system CPU usage. 4.7.0-rc3 4.7.0-rc3 mmotm-20160623 nodelru-v9 Minor Faults 362662 367360 Major Faults 1204 1143 Swap Ins 22 0 Swap Outs 2855 1029 DMA allocs 0 0 DMA32 allocs 31409797 28837521 Normal allocs 46611853 49231282 Movable allocs 0 0 Direct pages scanned 0 0 Kswapd pages scanned 40845270 40869088 Kswapd pages reclaimed 40830976 40855294 Direct pages reclaimed 0 0 Kswapd efficiency 99% 99% Kswapd velocity 11386.711 11369.769 Direct efficiency 100% 100% Direct velocity 0.000 0.000 Percentage direct scans 0% 0% Page writes by reclaim 2855 1029 Page writes file 0 0 Page writes anon 2855 1029 Page reclaim immediate 771 1628 Sector Reads 293312636 293536360 Sector Writes 18213568 18186480 Page rescued immediate 0 0 Slabs scanned 128257 132747 Direct inode steals 181 56 Kswapd inode steals 59 1131 It basically shows that kswapd was active at roughly the same rate in both kernels. There was also comparable slab scanning activity and direct reclaim was avoided in both cases. There appears to be a large difference in numbers of inodes reclaimed but the workload has few active inodes and is likely a timing artifact. stutter ------- stutter simulates a simple workload. One part uses a lot of anonymous memory, a second measures mmap latency and a third copies a large file. The primary metric is checking for mmap latency. stutter 4.7.0-rc4 4.7.0-rc4 mmotm-20160623 nodelru-v8 Min mmap 16.6283 ( 0.00%) 13.4258 ( 19.26%) 1st-qrtle mmap 54.7570 ( 0.00%) 34.9121 ( 36.24%) 2nd-qrtle mmap 57.3163 ( 0.00%) 46.1147 ( 19.54%) 3rd-qrtle mmap 58.9976 ( 0.00%) 47.1882 ( 20.02%) Max-90% mmap 59.7433 ( 0.00%) 47.4453 ( 20.58%) Max-93% mmap 60.1298 ( 0.00%) 47.6037 ( 20.83%) Max-95% mmap 73.4112 ( 0.00%) 82.8719 (-12.89%) Max-99% mmap 92.8542 ( 0.00%) 88.8870 ( 4.27%) Max mmap 1440.6569 ( 0.00%) 121.4201 ( 91.57%) Mean mmap 59.3493 ( 0.00%) 42.2991 ( 28.73%) Best99%Mean mmap 57.2121 ( 0.00%) 41.8207 ( 26.90%) Best95%Mean mmap 55.9113 ( 0.00%) 39.9620 ( 28.53%) Best90%Mean mmap 55.6199 ( 0.00%) 39.3124 ( 29.32%) Best50%Mean mmap 53.2183 ( 0.00%) 33.1307 ( 37.75%) Best10%Mean mmap 45.9842 ( 0.00%) 20.4040 ( 55.63%) Best5%Mean mmap 43.2256 ( 0.00%) 17.9654 ( 58.44%) Best1%Mean mmap 32.9388 ( 0.00%) 16.6875 ( 49.34%) This shows a number of improvements with the worst-case outlier greatly improved. Some of the vmstats are interesting 4.7.0-rc4 4.7.0-rc4 mmotm-20160623nodelru-v8 Swap Ins 163 502 Swap Outs 0 0 DMA allocs 0 0 DMA32 allocs 618719206 1381662383 Normal allocs 891235743 564138421 Movable allocs 0 0 Allocation stalls 2603 1 Direct pages scanned 216787 2 Kswapd pages scanned 50719775 41778378 Kswapd pages reclaimed 41541765 41777639 Direct pages reclaimed 209159 0 Kswapd efficiency 81% 99% Kswapd velocity 16859.554 14329.059 Direct efficiency 96% 0% Direct velocity 72.061 0.001 Percentage direct scans 0% 0% Page writes by reclaim 6215049 0 Page writes file 6215049 0 Page writes anon 0 0 Page reclaim immediate 70673 90 Sector Reads 81940800 81680456 Sector Writes 100158984 98816036 Page rescued immediate 0 0 Slabs scanned 1366954 22683 While this is not guaranteed in all cases, this particular test showed a large reduction in direct reclaim activity. It's also worth noting that no page writes were issued from reclaim context. This series is not without its hazards. There are at least three areas that I'm concerned with even though I could not reproduce any problems in that area. 1. Reclaim/compaction is going to be affected because the amount of reclaim is no longer targetted at a specific zone. Compaction works on a per-zone basis so there is no guarantee that reclaiming a few THP's worth page pages will have a positive impact on compaction success rates. 2. The Slab/LRU reclaim ratio is affected because the frequency the shrinkers are called is now different. This may or may not be a problem but if it is, it'll be because shrinkers are not called enough and some balancing is required. 3. The anon/file reclaim ratio may be affected. Pages about to be dirtied are distributed between zones and the fair zone allocation policy used to do something very similar for anon. The distribution is now different but not necessarily in any way that matters but it's still worth bearing in mind. VM statistic counters for reclaim decisions are zone-based. If the kernel is to reclaim on a per-node basis then we need to track per-node statistics but there is no infrastructure for that. The most notable change is that the old node_page_state is renamed to sum_zone_node_page_state. The new node_page_state takes a pglist_data and uses per-node stats but none exist yet. There is some renaming such as vm_stat to vm_zone_stat and the addition of vm_node_stat and the renaming of mod_state to mod_zone_state. Otherwise, this is mostly a mechanical patch with no functional change. There is a lot of similarity between the node and zone helpers which is unfortunate but there was no obvious way of reusing the code and maintaining type safety. Link: http://lkml.kernel.org/r/[email protected] Signed-off-by: Mel Gorman <[email protected]> Acked-by: Johannes Weiner <[email protected]> Acked-by: Vlastimil Babka <[email protected]> Cc: Rik van Riel <[email protected]> Cc: Vlastimil Babka <[email protected]> Cc: Johannes Weiner <[email protected]> Cc: Minchan Kim <[email protected]> Cc: Joonsoo Kim <[email protected]> Cc: Hillf Danton <[email protected]> Cc: Michal Hocko <[email protected]> Cc: Minchan Kim <[email protected]> Signed-off-by: Andrew Morton <[email protected]> Signed-off-by: Linus Torvalds <[email protected]>
1 parent a621184 commit 75ef718

File tree

7 files changed

+424
-76
lines changed

7 files changed

+424
-76
lines changed

‎drivers/base/node.c

+41-35
Original file line numberDiff line numberDiff line change
@@ -74,16 +74,16 @@ static ssize_t node_read_meminfo(struct device *dev,
7474
nid, K(i.totalram),
7575
nid, K(i.freeram),
7676
nid, K(i.totalram - i.freeram),
77-
nid, K(node_page_state(nid, NR_ACTIVE_ANON) +
78-
node_page_state(nid, NR_ACTIVE_FILE)),
79-
nid, K(node_page_state(nid, NR_INACTIVE_ANON) +
80-
node_page_state(nid, NR_INACTIVE_FILE)),
81-
nid, K(node_page_state(nid, NR_ACTIVE_ANON)),
82-
nid, K(node_page_state(nid, NR_INACTIVE_ANON)),
83-
nid, K(node_page_state(nid, NR_ACTIVE_FILE)),
84-
nid, K(node_page_state(nid, NR_INACTIVE_FILE)),
85-
nid, K(node_page_state(nid, NR_UNEVICTABLE)),
86-
nid, K(node_page_state(nid, NR_MLOCK)));
77+
nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_ANON) +
78+
sum_zone_node_page_state(nid, NR_ACTIVE_FILE)),
79+
nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_ANON) +
80+
sum_zone_node_page_state(nid, NR_INACTIVE_FILE)),
81+
nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_ANON)),
82+
nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_ANON)),
83+
nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_FILE)),
84+
nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_FILE)),
85+
nid, K(sum_zone_node_page_state(nid, NR_UNEVICTABLE)),
86+
nid, K(sum_zone_node_page_state(nid, NR_MLOCK)));
8787

8888
#ifdef CONFIG_HIGHMEM
8989
n += sprintf(buf + n,
@@ -117,31 +117,31 @@ static ssize_t node_read_meminfo(struct device *dev,
117117
"Node %d ShmemPmdMapped: %8lu kB\n"
118118
#endif
119119
,
120-
nid, K(node_page_state(nid, NR_FILE_DIRTY)),
121-
nid, K(node_page_state(nid, NR_WRITEBACK)),
122-
nid, K(node_page_state(nid, NR_FILE_PAGES)),
123-
nid, K(node_page_state(nid, NR_FILE_MAPPED)),
124-
nid, K(node_page_state(nid, NR_ANON_PAGES)),
120+
nid, K(sum_zone_node_page_state(nid, NR_FILE_DIRTY)),
121+
nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK)),
122+
nid, K(sum_zone_node_page_state(nid, NR_FILE_PAGES)),
123+
nid, K(sum_zone_node_page_state(nid, NR_FILE_MAPPED)),
124+
nid, K(sum_zone_node_page_state(nid, NR_ANON_PAGES)),
125125
nid, K(i.sharedram),
126-
nid, node_page_state(nid, NR_KERNEL_STACK) *
126+
nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK) *
127127
THREAD_SIZE / 1024,
128-
nid, K(node_page_state(nid, NR_PAGETABLE)),
129-
nid, K(node_page_state(nid, NR_UNSTABLE_NFS)),
130-
nid, K(node_page_state(nid, NR_BOUNCE)),
131-
nid, K(node_page_state(nid, NR_WRITEBACK_TEMP)),
132-
nid, K(node_page_state(nid, NR_SLAB_RECLAIMABLE) +
133-
node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
134-
nid, K(node_page_state(nid, NR_SLAB_RECLAIMABLE)),
128+
nid, K(sum_zone_node_page_state(nid, NR_PAGETABLE)),
129+
nid, K(sum_zone_node_page_state(nid, NR_UNSTABLE_NFS)),
130+
nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),
131+
nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK_TEMP)),
132+
nid, K(sum_zone_node_page_state(nid, NR_SLAB_RECLAIMABLE) +
133+
sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
134+
nid, K(sum_zone_node_page_state(nid, NR_SLAB_RECLAIMABLE)),
135135
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
136-
nid, K(node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
137-
nid, K(node_page_state(nid, NR_ANON_THPS) *
136+
nid, K(sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
137+
nid, K(sum_zone_node_page_state(nid, NR_ANON_THPS) *
138138
HPAGE_PMD_NR),
139-
nid, K(node_page_state(nid, NR_SHMEM_THPS) *
139+
nid, K(sum_zone_node_page_state(nid, NR_SHMEM_THPS) *
140140
HPAGE_PMD_NR),
141-
nid, K(node_page_state(nid, NR_SHMEM_PMDMAPPED) *
141+
nid, K(sum_zone_node_page_state(nid, NR_SHMEM_PMDMAPPED) *
142142
HPAGE_PMD_NR));
143143
#else
144-
nid, K(node_page_state(nid, NR_SLAB_UNRECLAIMABLE)));
144+
nid, K(sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)));
145145
#endif
146146
n += hugetlb_report_node_meminfo(nid, buf + n);
147147
return n;
@@ -160,25 +160,31 @@ static ssize_t node_read_numastat(struct device *dev,
160160
"interleave_hit %lu\n"
161161
"local_node %lu\n"
162162
"other_node %lu\n",
163-
node_page_state(dev->id, NUMA_HIT),
164-
node_page_state(dev->id, NUMA_MISS),
165-
node_page_state(dev->id, NUMA_FOREIGN),
166-
node_page_state(dev->id, NUMA_INTERLEAVE_HIT),
167-
node_page_state(dev->id, NUMA_LOCAL),
168-
node_page_state(dev->id, NUMA_OTHER));
163+
sum_zone_node_page_state(dev->id, NUMA_HIT),
164+
sum_zone_node_page_state(dev->id, NUMA_MISS),
165+
sum_zone_node_page_state(dev->id, NUMA_FOREIGN),
166+
sum_zone_node_page_state(dev->id, NUMA_INTERLEAVE_HIT),
167+
sum_zone_node_page_state(dev->id, NUMA_LOCAL),
168+
sum_zone_node_page_state(dev->id, NUMA_OTHER));
169169
}
170170
static DEVICE_ATTR(numastat, S_IRUGO, node_read_numastat, NULL);
171171

172172
static ssize_t node_read_vmstat(struct device *dev,
173173
struct device_attribute *attr, char *buf)
174174
{
175175
int nid = dev->id;
176+
struct pglist_data *pgdat = NODE_DATA(nid);
176177
int i;
177178
int n = 0;
178179

179180
for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++)
180181
n += sprintf(buf+n, "%s %lu\n", vmstat_text[i],
181-
node_page_state(nid, i));
182+
sum_zone_node_page_state(nid, i));
183+
184+
for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
185+
n += sprintf(buf+n, "%s %lu\n",
186+
vmstat_text[i + NR_VM_ZONE_STAT_ITEMS],
187+
node_page_state(pgdat, i));
182188

183189
return n;
184190
}

‎include/linux/mm.h

+5
Original file line numberDiff line numberDiff line change
@@ -933,6 +933,11 @@ static inline struct zone *page_zone(const struct page *page)
933933
return &NODE_DATA(page_to_nid(page))->node_zones[page_zonenum(page)];
934934
}
935935

936+
static inline pg_data_t *page_pgdat(const struct page *page)
937+
{
938+
return NODE_DATA(page_to_nid(page));
939+
}
940+
936941
#ifdef SECTION_IN_PAGE_FLAGS
937942
static inline void set_page_section(struct page *page, unsigned long section)
938943
{

‎include/linux/mmzone.h

+13
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,10 @@ enum zone_stat_item {
160160
NR_FREE_CMA_PAGES,
161161
NR_VM_ZONE_STAT_ITEMS };
162162

163+
enum node_stat_item {
164+
NR_VM_NODE_STAT_ITEMS
165+
};
166+
163167
/*
164168
* We do arithmetic on the LRU lists in various places in the code,
165169
* so it is important to keep the active lists LRU_ACTIVE higher in
@@ -267,6 +271,11 @@ struct per_cpu_pageset {
267271
#endif
268272
};
269273

274+
struct per_cpu_nodestat {
275+
s8 stat_threshold;
276+
s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS];
277+
};
278+
270279
#endif /* !__GENERATING_BOUNDS.H */
271280

272281
enum zone_type {
@@ -695,6 +704,10 @@ typedef struct pglist_data {
695704
struct list_head split_queue;
696705
unsigned long split_queue_len;
697706
#endif
707+
708+
/* Per-node vmstats */
709+
struct per_cpu_nodestat __percpu *per_cpu_nodestats;
710+
atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
698711
} pg_data_t;
699712

700713
#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)

‎include/linux/vmstat.h

+80-12
Original file line numberDiff line numberDiff line change
@@ -106,20 +106,38 @@ static inline void vm_events_fold_cpu(int cpu)
106106
zone_idx(zone), delta)
107107

108108
/*
109-
* Zone based page accounting with per cpu differentials.
109+
* Zone and node-based page accounting with per cpu differentials.
110110
*/
111-
extern atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
111+
extern atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS];
112+
extern atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS];
112113

113114
static inline void zone_page_state_add(long x, struct zone *zone,
114115
enum zone_stat_item item)
115116
{
116117
atomic_long_add(x, &zone->vm_stat[item]);
117-
atomic_long_add(x, &vm_stat[item]);
118+
atomic_long_add(x, &vm_zone_stat[item]);
119+
}
120+
121+
static inline void node_page_state_add(long x, struct pglist_data *pgdat,
122+
enum node_stat_item item)
123+
{
124+
atomic_long_add(x, &pgdat->vm_stat[item]);
125+
atomic_long_add(x, &vm_node_stat[item]);
118126
}
119127

120128
static inline unsigned long global_page_state(enum zone_stat_item item)
121129
{
122-
long x = atomic_long_read(&vm_stat[item]);
130+
long x = atomic_long_read(&vm_zone_stat[item]);
131+
#ifdef CONFIG_SMP
132+
if (x < 0)
133+
x = 0;
134+
#endif
135+
return x;
136+
}
137+
138+
static inline unsigned long global_node_page_state(enum node_stat_item item)
139+
{
140+
long x = atomic_long_read(&vm_node_stat[item]);
123141
#ifdef CONFIG_SMP
124142
if (x < 0)
125143
x = 0;
@@ -161,31 +179,44 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone,
161179
}
162180

163181
#ifdef CONFIG_NUMA
164-
165-
extern unsigned long node_page_state(int node, enum zone_stat_item item);
166-
182+
extern unsigned long sum_zone_node_page_state(int node,
183+
enum zone_stat_item item);
184+
extern unsigned long node_page_state(struct pglist_data *pgdat,
185+
enum node_stat_item item);
167186
#else
168-
169-
#define node_page_state(node, item) global_page_state(item)
170-
187+
#define sum_zone_node_page_state(node, item) global_page_state(item)
188+
#define node_page_state(node, item) global_node_page_state(item)
171189
#endif /* CONFIG_NUMA */
172190

173191
#define add_zone_page_state(__z, __i, __d) mod_zone_page_state(__z, __i, __d)
174192
#define sub_zone_page_state(__z, __i, __d) mod_zone_page_state(__z, __i, -(__d))
193+
#define add_node_page_state(__p, __i, __d) mod_node_page_state(__p, __i, __d)
194+
#define sub_node_page_state(__p, __i, __d) mod_node_page_state(__p, __i, -(__d))
175195

176196
#ifdef CONFIG_SMP
177197
void __mod_zone_page_state(struct zone *, enum zone_stat_item item, long);
178198
void __inc_zone_page_state(struct page *, enum zone_stat_item);
179199
void __dec_zone_page_state(struct page *, enum zone_stat_item);
180200

201+
void __mod_node_page_state(struct pglist_data *, enum node_stat_item item, long);
202+
void __inc_node_page_state(struct page *, enum node_stat_item);
203+
void __dec_node_page_state(struct page *, enum node_stat_item);
204+
181205
void mod_zone_page_state(struct zone *, enum zone_stat_item, long);
182206
void inc_zone_page_state(struct page *, enum zone_stat_item);
183207
void dec_zone_page_state(struct page *, enum zone_stat_item);
184208

209+
void mod_node_page_state(struct pglist_data *, enum node_stat_item, long);
210+
void inc_node_page_state(struct page *, enum node_stat_item);
211+
void dec_node_page_state(struct page *, enum node_stat_item);
212+
185213
extern void inc_zone_state(struct zone *, enum zone_stat_item);
214+
extern void inc_node_state(struct pglist_data *, enum node_stat_item);
186215
extern void __inc_zone_state(struct zone *, enum zone_stat_item);
216+
extern void __inc_node_state(struct pglist_data *, enum node_stat_item);
187217
extern void dec_zone_state(struct zone *, enum zone_stat_item);
188218
extern void __dec_zone_state(struct zone *, enum zone_stat_item);
219+
extern void __dec_node_state(struct pglist_data *, enum node_stat_item);
189220

190221
void quiet_vmstat(void);
191222
void cpu_vm_stats_fold(int cpu);
@@ -213,16 +244,34 @@ static inline void __mod_zone_page_state(struct zone *zone,
213244
zone_page_state_add(delta, zone, item);
214245
}
215246

247+
static inline void __mod_node_page_state(struct pglist_data *pgdat,
248+
enum node_stat_item item, int delta)
249+
{
250+
node_page_state_add(delta, pgdat, item);
251+
}
252+
216253
static inline void __inc_zone_state(struct zone *zone, enum zone_stat_item item)
217254
{
218255
atomic_long_inc(&zone->vm_stat[item]);
219-
atomic_long_inc(&vm_stat[item]);
256+
atomic_long_inc(&vm_zone_stat[item]);
257+
}
258+
259+
static inline void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
260+
{
261+
atomic_long_inc(&pgdat->vm_stat[item]);
262+
atomic_long_inc(&vm_node_stat[item]);
220263
}
221264

222265
static inline void __dec_zone_state(struct zone *zone, enum zone_stat_item item)
223266
{
224267
atomic_long_dec(&zone->vm_stat[item]);
225-
atomic_long_dec(&vm_stat[item]);
268+
atomic_long_dec(&vm_zone_stat[item]);
269+
}
270+
271+
static inline void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item)
272+
{
273+
atomic_long_dec(&pgdat->vm_stat[item]);
274+
atomic_long_dec(&vm_node_stat[item]);
226275
}
227276

228277
static inline void __inc_zone_page_state(struct page *page,
@@ -231,12 +280,26 @@ static inline void __inc_zone_page_state(struct page *page,
231280
__inc_zone_state(page_zone(page), item);
232281
}
233282

283+
static inline void __inc_node_page_state(struct page *page,
284+
enum node_stat_item item)
285+
{
286+
__inc_node_state(page_pgdat(page), item);
287+
}
288+
289+
234290
static inline void __dec_zone_page_state(struct page *page,
235291
enum zone_stat_item item)
236292
{
237293
__dec_zone_state(page_zone(page), item);
238294
}
239295

296+
static inline void __dec_node_page_state(struct page *page,
297+
enum node_stat_item item)
298+
{
299+
__dec_node_state(page_pgdat(page), item);
300+
}
301+
302+
240303
/*
241304
* We only use atomic operations to update counters. So there is no need to
242305
* disable interrupts.
@@ -245,7 +308,12 @@ static inline void __dec_zone_page_state(struct page *page,
245308
#define dec_zone_page_state __dec_zone_page_state
246309
#define mod_zone_page_state __mod_zone_page_state
247310

311+
#define inc_node_page_state __inc_node_page_state
312+
#define dec_node_page_state __dec_node_page_state
313+
#define mod_node_page_state __mod_node_page_state
314+
248315
#define inc_zone_state __inc_zone_state
316+
#define inc_node_state __inc_node_state
249317
#define dec_zone_state __dec_zone_state
250318

251319
#define set_pgdat_percpu_threshold(pgdat, callback) { }

‎mm/page_alloc.c

+8-2
Original file line numberDiff line numberDiff line change
@@ -4204,8 +4204,8 @@ void si_meminfo_node(struct sysinfo *val, int nid)
42044204
for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++)
42054205
managed_pages += pgdat->node_zones[zone_type].managed_pages;
42064206
val->totalram = managed_pages;
4207-
val->sharedram = node_page_state(nid, NR_SHMEM);
4208-
val->freeram = node_page_state(nid, NR_FREE_PAGES);
4207+
val->sharedram = sum_zone_node_page_state(nid, NR_SHMEM);
4208+
val->freeram = sum_zone_node_page_state(nid, NR_FREE_PAGES);
42094209
#ifdef CONFIG_HIGHMEM
42104210
for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) {
42114211
struct zone *zone = &pgdat->node_zones[zone_type];
@@ -5330,6 +5330,11 @@ static void __meminit setup_zone_pageset(struct zone *zone)
53305330
zone->pageset = alloc_percpu(struct per_cpu_pageset);
53315331
for_each_possible_cpu(cpu)
53325332
zone_pageset_init(zone, cpu);
5333+
5334+
if (!zone->zone_pgdat->per_cpu_nodestats) {
5335+
zone->zone_pgdat->per_cpu_nodestats =
5336+
alloc_percpu(struct per_cpu_nodestat);
5337+
}
53335338
}
53345339

53355340
/*
@@ -6033,6 +6038,7 @@ void __paginginit free_area_init_node(int nid, unsigned long *zones_size,
60336038
reset_deferred_meminit(pgdat);
60346039
pgdat->node_id = nid;
60356040
pgdat->node_start_pfn = node_start_pfn;
6041+
pgdat->per_cpu_nodestats = NULL;
60366042
#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
60376043
get_pfn_range_for_nid(nid, &start_pfn, &end_pfn);
60386044
pr_info("Initmem setup node %d [mem %#018Lx-%#018Lx]\n", nid,

0 commit comments

Comments
 (0)