runtime: memory of finished goroutines not freed

[ohhfeng]

Go version

go version go1.19 windows/amd64

Output of go env in your module/workspace:

Does not depend env

What did you do?

I have observed that after creating a large number of Goroutines, the memory does not seem to be released back to the system. Specifically, I suspect there might be an issue with the sched.gFree.noStack in proc.go, where the Goroutine structures are not being properly cleared, leading to potential memory leak.

What did you see happen?

The memory used by the finished Goroutines does not appear to be released, even when the Goroutines are no longer in use. This suggests a potential memory leak in the runtime's Goroutine management.

What did you expect to see?

I expected that after Goroutines finish their execution and are no longer needed, the memory they occupied would be returned to the system.

[septemhill]

I have observed that after creating a large number of Goroutines, the memory does not seem to be released back to the system.

In short period of time ? If so, it might related to #9869 ?

[seankhliao]

1.19 isn't a supported version anymore (currently 1.22 and 1.21).
how are you measuring memory?
do you have a reproducer?

[ohhfeng]

package main

import (
	"fmt"
	"net/http"
	_ "net/http/pprof"
	"runtime"
	"sync"
	"time"
)

func printMemStats() {
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	fmt.Println("`````````````````````````````````````````````")
	fmt.Println("Memory Allocation:")
	fmt.Println("Alloc:", m.Alloc)
	fmt.Println("TotalAlloc:", m.TotalAlloc)
	fmt.Println("HeapAlloc:", m.HeapAlloc)
	fmt.Println("HeapSys:", m.HeapSys)
	fmt.Println("HeapIdle:", m.HeapIdle)
	fmt.Println("HeapInuse:", m.HeapInuse)
	fmt.Println("HeapReleased:", m.HeapReleased)
	fmt.Println("HeapObjects:", m.HeapObjects)
	fmt.Println("runtime.NumGoroutine():", runtime.NumGoroutine())
}

func main() {
	go func() {
		fmt.Println(http.ListenAndServe("localhost:6060", nil))
	}()

	num := 1000000
	printMemStats()
	wg := sync.WaitGroup{}
	wg.Add(num)
	for i := 0; i < num; i++ {
		go func() {
			defer wg.Done()
			time.Sleep(time.Second)
		}()
	}
	wg.Wait()
	printMemStats()
	time.Sleep(time.Second * 5)
	runtime.GC()
	for {
		time.Sleep(time.Second * 10)
		printMemStats()
	}
}

The memory usage after creating one million goroutines is as follows.

The memory usage after a period of time is as follows.And then it never goes down again.

I think that the goroutines in sched.gFree.noStack have not been properly removed.

[randall77]

It is true that we never free a g structure once allocated. They get put in the allgs list and live there forevermore.
Freeing the gs could be done, I suppose, but would make unsynchronized access to allgs trickier because it would need to handle the shrinking case. And maybe there are other parts of the runtime that assume gs are never freed.

Which is to say, maybe we could do this, but it isn't trivial.

Is there a situation where this is actually causing a problem? Most servers, at least in steady state, would reuse these gs effectively.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)