httpsd memory leak in FortiOS 6.2.7 #62
Description
On my home Fortigate, a -61F running v6.2.7 build1190 (GA). I am entering conserve mode after ~12hr of polling at 15s intervals.
I am not observing this on v6.4.4 build1803 (GA) on a Fortigate-VM server.
fortigate (global) # get sys perf stat
[ ... ]
Memory: 1964908k total, 1140648k used (58.1%), 616532k free (31.4%), 207728k freeable (10.5%)
fortigate (global) # diagnose sys top
Run Time: 0 days, 6 hours and 40 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1918T, 602F
httpsd 250 S 0.9 6.0
httpsd 259 S 0.9 5.9
httpsd 262 S 0.0 6.3
httpsd 246 S 0.0 6.3
cmdbsvr 150 S 0.0 2.1
reportd 196 S 0.0 2.0
ipshelper 213 S < 0.0 1.8
cw_acd 218 S 0.0 1.7
miglogd 177 S 0.0 1.5
miglogd 242 S 0.0 1.4
miglogd 240 S 0.0 1.4
miglogd 239 S 0.0 1.4
[ ... ]
fortigate (global) # diag hard sysinfo memory
MemTotal: 1964908 kB
MemFree: 615768 kB
Buffers: 32632 kB
Cached: 276476 kB
SwapCached: 0 kB
Active: 723672 kB
Inactive: 133808 kB
Active(anon): 665168 kB
Inactive(anon): 22048 kB
Active(file): 58504 kB
Inactive(file): 111760 kB
Unevictable: 89536 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 380 kB
Writeback: 0 kB
AnonPages: 637980 kB
Mapped: 72888 kB
Shmem: 49292 kB
Slab: 195440 kB
SReclaimable: 17536 kB
SUnreclaim: 177904 kB
KernelStack: 2464 kB
PageTables: 16228 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 982452 kB
Committed_AS: 24775892 kB
VmallocTotal: 260046784 kB
VmallocUsed: 84992 kB
VmallocChunk: 259953168 kB
I then restart httpsd like this: fnsysctl killall httpsd. The result is:
fortigate (global) # get sys perf stat
[ ... ]
Memory: 1964908k total, 776172k used (39.5%), 979792k free (49.9%), 208944k freeable (10.6%)
fortigate (global) # diagnose sys top
Run Time: 0 days, 6 hours and 49 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1918T, 955F
miglogd 240 S 0.4 1.4
dnsproxy 217 S 0.4 0.9
cmdbsvr 150 S 0.0 2.1
reportd 196 S 0.0 1.9
ipshelper 213 S < 0.0 1.8
cw_acd 218 S 0.0 1.7
httpsd 2235 S 0.0 1.5
miglogd 177 S 0.0 1.5
miglogd 242 S 0.0 1.4
miglogd 243 S 0.0 1.4
miglogd 239 S 0.0 1.4
httpsd 2237 S 0.0 1.4
httpsd 2236 S 0.0 1.3
httpsd 2234 S 0.0 1.2
[ ... ]
fortigate (global) # diag hard sysinfo memory
MemTotal: 1964908 kB
MemFree: 980336 kB
Buffers: 32972 kB
Cached: 277820 kB
SwapCached: 0 kB
Active: 358048 kB
Inactive: 134784 kB
Active(anon): 299280 kB
Inactive(anon): 21972 kB
Active(file): 58768 kB
Inactive(file): 112812 kB
Unevictable: 89536 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 488 kB
Writeback: 0 kB
AnonPages: 271596 kB
Mapped: 73024 kB
Shmem: 49660 kB
Slab: 196140 kB
SReclaimable: 17604 kB
SUnreclaim: 178536 kB
KernelStack: 2496 kB
PageTables: 15772 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 982452 kB
Committed_AS: 25006324 kB
VmallocTotal: 260046784 kB
VmallocUsed: 84992 kB
VmallocChunk: 259949872 kB
Log for httpsd in debug level -1 attached but I cannot seem to detect any odd things.
httpsd.log
Given that I am not observing this leak on my Fortigate-VM instance, I am thinking this memory leak is probably one of the following:
- Bug in 6.2.7
- Bug on all current FortiOS on 61F
- Bug on all current FortiOS under some specific circumstances
I should be able to upgrade to 6.4.4 soon enough and give that a shot on my 61F. I might also be able to spin up a Fortigate-VM 6.2.7 and see if that has the same behavior.
I have not reported this issue to Fortinet as of yet.
Maybe interesting is that the memory usage is reported to belong to the main VDOM which is my traffic forwarding VDOM, not the management one (root).
