memory usage of s3 upload manager for Go 1.13.5 and 1.13.6

[weaselsnak]

Version of AWS SDK for Go?

v1.28.0

Version of Go (go version)?

1.13.5, 1.13.6

What issue did you see?

Excessive memory consumption when concurrently uploading to s3 using the s3 upload manager. The program runs out of memory and crashes when uploading from an AWS EC2 linux instance with 1 GB RAM. The program does not crash or leak memory when running with Go 1.12.14.

Profiling Go 1.12.14 on instance: The top nodes don't contain any memory usage related to aws-sdk-go

Profiling Go 1.13.6 on instance: Program crashes (as it runs out of memory) and unable to see profiling output, however running the program locally (16 GB RAM) I can see all the memory is consumed by the s3 upload processes unlike the above.

(pprof) top
Showing nodes accounting for 266243.44kB, 99.40% of 267842.77kB total
Dropped 501 nodes (cum <= 1339.21kB)
Showing top 10 nodes out of 11
flat flat% sum% cum cum%
266240.12kB 99.40% 99.40% 266240.12kB 99.40% github.com/aws/aws-sdk-go/service/s3/s3manager.newPartPool.func1
1kB 0.00037% 99.40% 235524.98kB 87.93% github.com/aws/aws-sdk-go/service/s3/s3manager.(*multiuploader).upload
0.59kB 0.00022% 99.40% 266246.66kB 99.40% github.com/aws/aws-sdk-go/service/s3/s3manager.Uploader.UploadWithContext
0.38kB 0.00014% 99.40% 266245.95kB 99.40% github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).upload
0.09kB 3.5e-05% 99.40% 266241.81kB 99.40% github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).nextReader
0 0% 99.40% 266241.62kB 99.40% github.com/aws/aws-sdk-go/service/s3/s3manager.(*partPool).Get
0 0% 99.40% 266246.66kB 99.40% github.com/aws/aws-sdk-go/service/s3/s3manager.Uploader.Upload

Steps to reproduce

Run the code below using Go 1.13.6 from an AWS EC2 instance (I did this with 10 uploads each around 200 MB in size)

package main

import (
	"io"
	"log"
	"math/rand"
	"net/http"
	"net/url"
	"os"
	"sync"
	"time"

	_ "net/http/pprof"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3/s3manager"
	"github.com/pkg/profile"
)

func (s s3uploader) uploadToS3(path string, res io.Reader) error {
	_, err := s.u.Upload(&s3manager.UploadInput{
		Bucket: aws.String(os.Getenv("S3_STORAGE_BUCKET")),
		Key:    aws.String(path),
		Body:   res,
	})
	if err != nil {
		return err
	}
	return nil
}

func (s s3uploader) syncOne(source string) error {
	u, err := url.Parse(source)
	if err != nil {
		return err
	}
	var res *http.Response
	res, err = http.Get(source)
	if err != nil {
		return err
	}
	defer res.Body.Close()
	log.Printf("uploading: %s", source)
	if err := s.uploadToS3(u.Path, res.Body); err != nil {
		return err
	}
	return nil
}

func (s s3uploader) sync(urls []string, concurrency int) error {
	work := make(chan string)
	var wg sync.WaitGroup
	for i := 0; i < concurrency; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for url := range work {
				if err := s.syncOne(url); err != nil {
					time.Sleep(time.Millisecond*time.Duration(rand.Intn(5000)) + time.Second)
					log.Printf("Retrying url: %s", url)
				}
			}
		}()
	}

	for _, url := range urls {
		work <- url
	}
	close(work)
	wg.Wait()
	return nil
}

type s3uploader struct {
	u *s3manager.Uploader
}

func main() {
	go func() {
		log.Println(http.ListenAndServe("localhost:6060", nil))
	}()
	defer profile.Start(profile.MemProfile).Stop()
	conf := aws.Config{Region: aws.String(os.Getenv("AWS_REGION"))}
	s := session.New(&conf)
	uploader := s3manager.NewUploader(s, func(u *s3manager.Uploader) {
		u.PartSize = 10 * 1024 * 1024 // 10MB per part
	})
	var ul s3uploader
	ul.u = uploader

	const concurrency = 5
	if err := ul.sync(os.Args[1:], concurrency); err != nil {
		log.Printf("  ...syncing failed: %s", err)
		return
	}

}

[italolelis]

We are also experiencing this. We have the same scenario as described above. We are uploading about 10 documents in parallel and getting a much much higher memory consumption.

Happy to help in any way we can to solve this.

[jasdel]

Thanks for reaching out and creating this issue. In local testing with Go 1.13.6 using the sample code I'm seeing ~250MB allocated for the program's run. I think this is expected though. The S3 Upload Manager's concurrence is per part of an individual object upload, not aggregated across multiple concurrent calls to Upload. This would explain the ~250MB memory. 5 concurrent calls to Upload, with 5 parts of that file uploaded concurrently per upload, and a part size of 10MB, (5 * 5 * 10MB)

I think go1.12 was not correctly keeping track of the sync.Pool's allocations. Prior to Go 1.13 sync.Pool's allocations had a high chance of being GC'ed and could actually cause a higher amount of memory allocated over the life of the program. But when running pprof's traces github.com/aws/aws-sdk-go/service/s3/s3manager.newPartPool.func1 on Go 1.12.14 we see it doesn't keep track of any allocated memory by the sync.Pool. Whereas Go 1.13.6 does.

Go 1.12.14:

     bytes:  5MB
         0   github.com/aws/aws-sdk-go/service/s3/s3manager.newPartPool.func1

Go 1.13.6:

     bytes:  5MB
     175MB   github.com/aws/aws-sdk-go/service/s3/s3manager.newPartPool.func1

With that said I'd like to learn more about the out of memory exception that you're application received. Are you able to provide anymore background on the conditions leading up to the out of memory exception? Such as configuration, size of files, concurrency, etc. In addition was the S3 Upload Manager shared between all workers performing the upload?

[MaerF0x0]

We've also encountered excessive memory consumption from buffering from a faster source than the upload. We've had to restrict it down using a io.Pipe that creates upstream backpressure for whatever the source is.

Eg: imagine your local SSD reads at 500MB/s but your uplink is only 125MB/s then you'll pull most of the file into memory before the manager can upload it.