go vet using more memory than all the other checks combined

[ncw]

I was trying to work out why golangci-lint is using 5GB of memory, so I disabled the linters one at a time. I discovered disabling the govet linter reduces the memory from 5,335,264k to 2,665,052k so just govet on its own is using 2.5GB of RAM.

The measurements were taken using time -v golangci-lint and taking the Maximum resident set size (kbytes)

I think this might be indicative of a problem. I tried with the master version and this is improved from 5,335,264k to 3,906,860k which is better but not best!

This is on https://github.com/ncw/rclone

1. Version of golangci-lint: golangci-lint --version (or git commit if you don't use binary distribution)

$ golangci-lint --version
golangci-lint has version 1.17.1 built from 4ba2155 on 2019-06-10T09:06:49Z

2. Config file: cat .golangci.yml

# golangci-lint configuration options

linters:
  enable: # numbers show k of RAM used for each option on its own
    - deadcode    # 1130400
    - errcheck    # 1155060
    - goimports   #  109252
    - golint      #  181900
    - ineffassign #  104260
    - structcheck # 1162180
    - varcheck    # 1157420
    - govet       # 3173336
    - unconvert   # 1149696
    #- prealloc
    #- maligned
  disable-all: true

issues:
  # Enable some lints excluded by default
  exclude-use-default: false

  # Maximum issues count per one linter. Set to 0 to disable. Default is 50.
  max-per-linter: 0

  # Maximum count of issues with the same text. Set to 0 to disable. Default is 3.
  max-same-issues: 0

3. Go environment: go version && go env

$ go version && go env
go version go1.12.6 linux/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ncw/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ncw/go"
GOPROXY=""
GORACE=""
GOROOT="/opt/go/go1.12"
GOTMPDIR=""
GOTOOLDIR="/opt/go/go1.12/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build142887997=/tmp/go-build -gno-record-gcc-switches"

4. Verbose output of running: golangci-lint run -v

$ golangci-lint run -v
INFO [config_reader] Config search paths: [./ /home/ncw/go/src/github.com/ncw/rclone /home/ncw/go/src/github.com/ncw /home/ncw/go/src/github.com /home/ncw/go/src /home/ncw/go /home/ncw /home /] 
INFO [config_reader] Used config file .golangci.yml 
INFO [lintersdb] Active 9 linters: [deadcode errcheck goimports golint govet ineffassign structcheck unconvert varcheck] 
INFO [loader] Go packages loading at mode load types and syntax took 4.128727585s 
INFO [loader] SSA repr building timing: packages building 39.410238ms, total 666.606199ms 
INFO [runner] worker.4 took 5.823383433s with stages: govet: 5.134636322s, varcheck: 250.451169ms, deadcode: 151.105242ms, errcheck: 102.373427ms, structcheck: 94.187916ms, unconvert: 90.583216ms 
INFO [runner] worker.3 took 5.866044967s with stages: ineffassign: 5.866036553s 
INFO [runner] worker.2 took 8.265849768s with stages: goimports: 8.265831935s 
INFO [runner] worker.1 took 10.925640836s with stages: golint: 10.925627368s 
INFO [runner] Workers idle times: #2: 2.64882149s, #3: 5.03845539s, #4: 5.101301831s 
INFO [runner] Issues before processing: 7, after processing: 0 
INFO [runner] processing took 3.091587ms with stages: nolint: 2.481695ms, autogenerated_exclude: 234.869µs, identifier_marker: 182.104µs, skip_dirs: 118.612µs, cgo: 24.374µs, path_prettifier: 16.986µs, max_same_issues: 10.871µs, skip_files: 8.903µs, filename_unadjuster: 6.413µs, diff: 2.353µs, exclude: 912ns, exclude-rules: 694ns, max_from_linter: 651ns, uniq_by_line: 624ns, max_per_file_from_linter: 560ns, path_shortener: 520ns, source_code: 446ns 
INFO File cache stats: 0 entries of total size 0B 
INFO Memory: 74 samples, avg is 2466.7MB, max is 5554.9MB 
INFO Execution took 15.785717043s                 

[howardjohn]

+1, removing govet brings istio/istio from 32gb to 9gb usage

[jirfag]

Hi, thank you! I’m actively reducing govet memory usage now. Please, try the latest version from master

[howardjohn]

Great, thanks!

[jirfag]

@howardjohn @ncw
I've significantly reduced memory usage of staticcheck, gosimple, stylecheck, govet and bodyclose.
Please, try the latest version from the master branch.

[howardjohn]

@jirfag

1.18:

INFO File cache stats: 1883 entries of total size 14.5MiB
INFO Memory: 247 samples, avg is 15160.7MB, max is 23435.9MB
INFO Execution took 47.142363018s

master/1.19:

INFO File cache stats: 1947 entries of total size 15.1MiB
INFO Memory: 232 samples, avg is 19879.1MB, max is 35973.2MB
INFO Execution took 1m8.966628422s

It seems to actually be a bit worse in master for some reason... maybe due to new linters? In 1.18 I have no lint errors but with 1.19 there are bunch, presumably from new linters?

When I run with default config though 1.18=INFO Memory: 83 samples, avg is 3017.3MB, max is 5988.1MB and 1.19=INFO Memory: 53 samples, avg is 1958.0MB, max is 4914.1MB so there is some improvement there

Here is our config:

service:
  # When updating this, also update the version stored in docker/build-tools/Dockerfile in the istio/tools repo.
  golangci-lint-version: 1.18.x # use the fixed version to not introduce new linters unexpectedly
run:
  # timeout for analysis, e.g. 30s, 5m, default is 1m
  deadline: 20m

  # which dirs to skip: they won't be analyzed;
  # can use regexp here: generated.*, regexp is applied on full path;
  # default value is empty list, but next dirs are always skipped independently
  # from this option's value:
  #   	vendor$, third_party$, testdata$, examples$, Godeps$, builtin$
  skip-dirs:
    - genfiles$
    - vendor$

  # which files to skip: they will be analyzed, but issues from them
  # won't be reported. Default value is empty list, but there is
  # no need to include all autogenerated files, we confidently recognize
  # autogenerated files. If it's not please let us know.
  skip-files:
    - ".*\\.pb\\.go"
    - ".*\\.gen\\.go"

linters:
  enable-all: true
  disable:
    - depguard
    - dupl
    - gochecknoglobals
    - gochecknoinits
    - goconst
    - gocyclo
    - gosec
    - nakedret
    - prealloc
    - scopelint
    - funlen
    - bodyclose
  fast: false

linters-settings:
  errcheck:
    # report about not checking of errors in type assetions: `a := b.(MyStruct)`;
    # default is false: such cases aren't reported by default.
    check-type-assertions: false

    # report about assignment of errors to blank identifier: `num, _ := strconv.Atoi(numStr)`;
    # default is false: such cases aren't reported by default.
    check-blank: false
  govet:
    # report about shadowed variables
    check-shadowing: false
  golint:
    # minimal confidence for issues, default is 0.8
    min-confidence: 0.0
  gofmt:
    # simplify code: gofmt with `-s` option, true by default
    simplify: true
  goimports:
    # put imports beginning with prefix after 3rd-party packages;
    # it's a comma-separated list of prefixes
    local-prefixes: istio.io/
  maligned:
    # print struct with more effective memory layout or not, false by default
    suggest-new: true
  misspell:
    # Correct spellings using locale preferences for US or UK.
    # Default is to use a neutral variety of English.
    # Setting locale to US will correct the British spelling of 'colour' to 'color'.
    locale: US
    ignore-words:
    - cancelled
  lll:
    # max line length, lines longer will be reported. Default is 120.
    # '\t' is counted as 1 character by default, and can be changed with the tab-width option
    line-length: 160
    # tab width in spaces. Default to 1.
    tab-width: 1
  unused:
    # treat code as a program (not a library) and report unused exported identifiers; default is false.
    # XXX: if you enable this setting, unused will report a lot of false-positives in text editors:
    # if it's called for subdir of a project it can't find funcs usages. All text editor integrations
    # with golangci-lint call it on a directory with the changed file.
    check-exported: false
  unparam:
    # call graph construction algorithm (cha, rta). In general, use cha for libraries,
    # and rta for programs with main packages. Default is cha.
    algo: cha

    # Inspect exported functions, default is false. Set to true if no external program/library imports your code.
    # XXX: if you enable this setting, unparam will report a lot of false-positives in text editors:
    # if it's called for subdir of a project it can't find external interfaces. All text editor integrations
    # with golangci-lint call it on a directory with the changed file.
    check-exported: false
  gocritic:
    enabled-checks:
      - appendCombine
      - argOrder
      - assignOp
      - badCond
      - boolExprSimplify
      - builtinShadow
      - captLocal
      - caseOrder
      - codegenComment
      - commentedOutCode
      - commentedOutImport
      - defaultCaseOrder
      - deprecatedComment
      - docStub
      - dupArg
      - dupBranchBody
      - dupCase
      - dupSubExpr
      - elseif
      - emptyFallthrough
      - equalFold
      - flagDeref
      - flagName
      - hexLiteral
      - indexAlloc
      - initClause
      - methodExprCall
      - nilValReturn
      - octalLiteral
      - offBy1
      - rangeExprCopy
      - regexpMust
      - sloppyLen
      - stringXbytes
      - switchTrue
      - typeAssertChain
      - typeSwitchVar
      - typeUnparen
      - underef
      - unlambda
      - unnecessaryBlock
      - unslice
      - valSwap
      - weakCond

      # Unused
      # - yodaStyleExpr
      # - appendAssign
      # - commentFormatting
      # - emptyStringTest
      # - exitAfterDefer
      # - ifElseChain
      # - hugeParam
      # - importShadow
      # - nestingReduce
      # - paramTypeCombine
      # - ptrToRefParam
      # - rangeValCopy
      # - singleCaseSwitch
      # - sloppyReassign
      # - unlabelStmt
      # - unnamedResult
      # - wrapperFunc

issues:
  # List of regexps of issue texts to exclude, empty list by default.
  # But independently from this option we use default exclude patterns,
  # it can be disabled by `exclude-use-default: false`. To list all
  # excluded by default patterns execute `golangci-lint run --help`
  exclude:
    - composite literal uses unkeyed fields

  exclude-rules:
    # Exclude some linters from running on test files.
    - path: _test\.go$|^tests/|^samples/
      linters:
        - errcheck
        - maligned

  # Independently from option `exclude` we use default exclude patterns,
  # it can be disabled by this option. To list all
  # excluded by default patterns execute `golangci-lint run --help`.
  # Default value for this option is true.
  exclude-use-default: true

  # Maximum issues count per one linter. Set to 0 to disable. Default is 50.
  max-per-linter: 0

  # Maximum count of issues with the same text. Set to 0 to disable. Default is 3.
  max-same-issues: 0

Let me know if there is any other info that would be useful. Thanks!

[jirfag]

I've testes on rclone/rclone:

# golangci-lint v1.19.0
~/go/src/github.com/rclone/rclone (master)$ rm -rf /Users/denis/Library/Caches/golangci-lint/; golangci-lint run -v --deadline=10m
INFO [config_reader] Config search paths: [./ /Users/denis/go/src/github.com/rclone/rclone /Users/denis/go/src/github.com/rclone /Users/denis/go/src/github.com /Users/denis/go/src /Users/denis/go /Users/denis /Users /]
INFO [config_reader] Used config file .golangci.yml
INFO [lintersdb] Active 9 linters: [deadcode errcheck goimports golint govet ineffassign structcheck unconvert varcheck]
INFO [loader] Go packages loading at mode 1023 (types|exports_file|files|imports|name|types_sizes|compiled_files|deps|syntax|types_info) took 7.147402332s
INFO [runner] worker.2 took 2.357788092s with stages: ineffassign: 1.634795609s, varcheck: 238.025179ms, deadcode: 188.148175ms, structcheck: 153.077733ms, errcheck: 91.743593ms, unconvert: 51.953024ms
INFO [runner] worker.4 took 3.712740745s with stages: govet: 3.712723858s
INFO [runner] worker.1 took 4.941617202s with stages: goimports: 4.941605858s
INFO [runner] worker.3 took 8.331572595s with stages: golint: 8.33153314s
INFO [runner] Workers idle times: #1: 3.386471958s, #2: 5.969868918s, #4: 4.618470137s
INFO [runner] Issues before processing: 5, after processing: 0
INFO [runner] Processors filtering stat (out/in): filename_unadjuster: 5/5, skip_files: 5/5, skip_dirs: 5/5, nolint: 0/5, autogenerated_exclude: 5/5, exclude: 5/5, cgo: 5/5, path_prettifier: 5/5, identifier_marker: 5/5, exclude-rules: 5/5
INFO [runner] processing took 13.077643ms with stages: nolint: 12.298243ms, autogenerated_exclude: 358.833µs, identifier_marker: 277.3µs, skip_dirs: 96.46µs, path_prettifier: 17.744µs, cgo: 13.602µs, filename_unadjuster: 7.532µs, max_same_issues: 2.21µs, uniq_by_line: 1.021µs, path_shortener: 811ns, source_code: 741ns, diff: 677ns, max_from_linter: 640ns, exclude: 572ns, skip_files: 498ns, exclude-rules: 381ns, max_per_file_from_linter: 378ns
INFO File cache stats: 0 entries of total size 0B
INFO Memory: 100 samples, avg is 961.7MB, max is 1942.4MB
INFO Execution took 15.659747905s

# golangci-lint v1.18.0
~/go/src/github.com/rclone/rclone (master)$ rm -rf /Users/denis/Library/Caches/golangci-lint/; golangci-lint run -v --deadline=10m
INFO [config_reader] Config search paths: [./ /Users/denis/go/src/github.com/rclone/rclone /Users/denis/go/src/github.com/rclone /Users/denis/go/src/github.com /Users/denis/go/src /Users/denis/go /Users/denis /Users /]
INFO [config_reader] Used config file .golangci.yml
INFO [lintersdb] Active 9 linters: [deadcode errcheck goimports golint govet ineffassign structcheck unconvert varcheck]
INFO [loader] Go packages loading at mode 991 (compiled_files|deps|syntax|types|types_sizes|files|imports|name|types_info) took 6.679540583s
INFO [loader] SSA repr building timing: packages building 59.940705ms, total 1.066718915s
INFO [runner] worker.2 took 9.130722297s with stages: govet: 8.507241807s, varcheck: 279.994423ms, deadcode: 155.169048ms, structcheck: 89.096583ms, unconvert: 51.354186ms, errcheck: 47.822878ms
INFO [runner] worker.4 took 11.660717482s with stages: ineffassign: 11.660709067s
INFO [runner] worker.3 took 14.911251531s with stages: goimports: 14.911228063s
INFO [runner] worker.1 took 19.345719863s with stages: golint: 19.345689489s
INFO [runner] Workers idle times: #2: 10.214754589s, #3: 4.429471802s, #4: 7.679749121s
INFO [runner] Issues before processing: 5, after processing: 0
INFO [runner] processing took 20.36669ms with stages: nolint: 10.633588ms, identifier_marker: 5.002473ms, autogenerated_exclude: 4.115974ms, cgo: 312.078µs, skip_dirs: 254.784µs, path_prettifier: 34.395µs, filename_unadjuster: 6.103µs, max_same_issues: 2.205µs, uniq_by_line: 944ns, diff: 663ns, max_from_linter: 652ns, source_code: 649ns, exclude: 530ns, path_shortener: 459ns, exclude-rules: 451ns, skip_files: 426ns, max_per_file_from_linter: 316ns
INFO File cache stats: 0 entries of total size 0B
INFO Memory: 130 samples, avg is 2194.6MB, max is 5355.2MB
INFO Execution took 28.362725585s

But I didn't test on the config with enable-all: true yet

[ncw]

That is a massive improvement , thank you and well done :-)

# golangci-lint v1.19.0
INFO Memory: 100 samples, avg is 961.7MB, max is 1942.4MB

# golangci-lint v1.18.0
INFO Memory: 130 samples, avg is 2194.6MB, max is 5355.2MB

[jirfag]

anyway, let's continue in #337