lukas-brenning · lukas-brenning · Jan 13, 2021 · Nov 4, 2020 · Dec 11, 2020 · Dec 14, 2020
diff --git a/LANGUAGES.md b/LANGUAGES.md
@@ -33,7 +33,6 @@ C# (cs,csx)
 C++ (cc,cpp,cxx,c++,pcc)
 C++ Header (hh,hpp,hxx,inl,ipp)
 Cabal (cabal)
-Cargo Lock (cargo.lock)
 Cassius (cassius)
 Ceylon (ceylon)
 Clojure (clj,cljc)
@@ -85,7 +84,7 @@ gitignore (.gitignore)
 GLSL (vert,tesc,tese,geom,frag,comp)
 GN (gn,gni)
 Go (go)
-Go Template (tmpl)
+Go Template (tmpl,gohtml,gotxt)
 Gradle (gradle)
 Groovy (groovy,grt,gtpl,gvy)
 HAML (haml)

diff --git a/README.md b/README.md
@@ -23,7 +23,7 @@ Dual-licensed under MIT or the [UNLICENSE](http://unlicense.org).
 
 #### Go Get
 
-If you are comfortable using Go and have >= 1.10 installed:
+If you are comfortable using Go and have >= 1.13 installed:
 
 `$ go get -u github.com/boyter/scc/`
 
@@ -95,12 +95,12 @@ For performance see the [Performance](https://github.com/boyter/scc#performance)
 
 Other similar projects,
 
- - [cloc](https://github.com/AlDanial/cloc) the original sloc counter
+ - [SLOCCount](https://www.dwheeler.com/sloccount/) the original sloc counter
+ - [cloc](https://github.com/AlDanial/cloc), inspired by SLOCCount; implemented in Perl for portability
  - [gocloc](https://github.com/hhatto/gocloc) a sloc counter in Go inspired by tokei
  - [loc](https://github.com/cgag/loc) rust implementation similar to tokei but often faster
  - [loccount](https://gitlab.com/esr/loccount) Go implementation written and maintained by ESR
  - [ployglot](https://github.com/vmchale/polyglot) ATS sloc counter
- - [sloccount](https://www.dwheeler.com/sloccount/) written as a faster cloc
  - [tokei](https://github.com/XAMPPRocky/tokei) fast, accurate and written in rust
 
 Interesting reading about other code counting projects tokei, loc, polyglot and loccount
@@ -178,7 +178,6 @@ Command line usage of `scc` is designed to be as simple as possible.
 Full details can be found in `scc --help` or `scc -h`. Note that the below reflects the state of master not a release.
 
 ```
-$ scc -h
 Sloc, Cloc and Code. Count lines of code in a directory with complexity estimation.
 Version 3.0.0 (beta)
 Ben Boyter <[email protected]> + Contributors
@@ -195,7 +194,7 @@ Flags:
       --debug                       enable debug output
       --exclude-dir strings         directories to exclude (default [.git,.hg,.svn])
       --file-gc-count int           number of files to parse before turning the GC on (default 10000)
-  -f, --format string               set output format [tabular, wide, json, csv, cloc-yaml, html, html-table, sql, sql-insert] (default "tabular")
+  -f, --format string               set output format [tabular, wide, json, csv, csv-stream, cloc-yaml, html, html-table, sql, sql-insert] (default "tabular")
       --format-multi string         have multiple format output overriding --format [e.g. tabular:stdout,csv:file.csv,json:file.json]
       --gen                         identify generated files
       --generated-markers strings   string markers in head of generated files (default [do not edit])
@@ -285,7 +284,7 @@ You can also run against multiple files or directories `scc directory1 directory
 ### Interesting Use Cases
 
 Used inside Intel Nemu Hypervisor to track code changes between revisions https://github.com/intel/nemu/blob/topic/virt-x86/tools/cloc-change.sh#L9
-Appears to also be used inside both http://codescoop.com/ and https://pinpoint.com/
+Appears to also be used inside both http://codescoop.com/ https://pinpoint.com/ https://github.com/chaoss/grimoirelab-graal
 
 It also is used to count code and guess language types in https://searchcode.com/ which makes it one of the most frequently run code counters in the world. 
 
@@ -383,7 +382,7 @@ Note that in all cases if the remap rule does not apply normal #! rules will app
 
 By default `scc` will output to the console. However you can produce output in other formats if you require.
 
-The different options are `tabular, wide, json, csv, cloc-yaml, html, html-table, sql, sql-insert`. 
+The different options are `tabular, wide, json, csv, csv-stream, cloc-yaml, html, html-table, sql, sql-insert`. 
 
 Note that you can write `scc` output to disk using the `-o, --output` option. This allows you to specify a file to
 write your output to. For example `scc -f html -o output.html` will run `scc` against the current directory, and output
@@ -422,6 +421,13 @@ CSV as an option is good for importing into a spreadsheet for analysis.
 Note that this format will give you the byte size of every file it `scc` reads allowing you to get a breakdown of the
 number of bytes processed.
 
+#### CSV-Stream
+
+csv-stream is an option useful for processing very large repositories where you are likely to run into memory issues. It's output format is 100% the same as CSV. 
+
+Note that you should not use this with the `format-multi` option as it will always print to standard output, and because of how it works will negate the memory saving it normally gains.
+savings that this option provides. Note that there is no sort applied with this option. 
+
 #### cloc-yaml 
 
 Is a drop in replacement for cloc using its yaml output option. This is quite often used for passing into other 
@@ -694,10 +700,35 @@ Run go build for windows and linux then the following in linux, keep in mind nee
 
 ```
 GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w" && zip -r9 scc-2.13.0-x86_64-apple-darwin.zip scc
+GOOS=darwin GOARCH=arm64 go build -ldflags="-s -w" && zip -r9 scc-2.13.0-arm64-apple-darwin.zip scc
 GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" && zip -r9 scc-2.13.0-x86_64-pc-windows.zip scc.exe
 GOOS=windows GOARCH=386 go build -ldflags="-s -w" && zip -r9 scc-2.13.0-i386-pc-windows.zip scc.exe
 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" && zip -r9 scc-2.13.0-x86_64-unknown-linux.zip scc
 GOOS=linux GOARCH=386 go build -ldflags="-s -w" && zip -r9 scc-2.13.0-i386-unknown-linux.zip scc
+GOOS=linux GOARCH=arm64 go build -ldflags="-s -w" && zip -r9 scc-2.13.0-arm64-unknown-linux.zip scc
+```
+
+### Containers
+
+Note if you plan to run `scc` in Alpine containers you will need to build with CGO_ENABLED=0.
+
+See the below dockerfile as an example on how to achieve this based on this issue https://github.com/boyter/scc/issues/208
+
+```
+FROM golang as scc-get
+
+ENV GOOS=linux \
+GOARCH=amd64 \
+CGO_ENABLED=0
+
+ARG VERSION
+RUN git clone --branch $VERSION --depth 1 https://github.com/boyter/scc
+WORKDIR /go/scc
+RUN go build -ldflags="-s -w"
+
+FROM alpine
+COPY --from=scc-get /go/scc/scc /bin/
+ENTRYPOINT ["scc"]
 ```
 
 ### Badges (beta)

diff --git a/SCC-OUTPUT-REPORT.html b/SCC-OUTPUT-REPORT.html
@@ -12,12 +12,12 @@
 	<tbody><tr>
 		<th>Go</th>
 		<th>34</th>
-		<th>8208</th>
-		<th>1340</th>
-		<th>349</th>
-		<th>6519</th>
-		<th>1352</th>
-		<th>325373</th>
+		<th>8332</th>
+		<th>1353</th>
+		<th>361</th>
+		<th>6618</th>
+		<th>1365</th>
+		<th>328465</th>
 	</tr><tr>
 		<th>Java</th>
 		<th>24</th>
@@ -36,6 +36,15 @@
 		<th>175</th>
 		<th>53</th>
 		<th>796847</th>
+	</tr><tr>
+		<th>Markdown</th>
+		<th>11</th>
+		<th>1197</th>
+		<th>286</th>
+		<th>0</th>
+		<th>911</th>
+		<th>0</th>
+		<th>49244</th>
 	</tr><tr>
 		<th>Python</th>
 		<th>9</th>
@@ -45,15 +54,6 @@
 		<th>310</th>
 		<th>32</th>
 		<th>10072</th>
-	</tr><tr>
-		<th>Markdown</th>
-		<th>8</th>
-		<th>1190</th>
-		<th>283</th>
-		<th>0</th>
-		<th>907</th>
-		<th>0</th>
-		<th>48508</th>
 	</tr><tr>
 		<th>CSS</th>
 		<th>5</th>
@@ -93,12 +93,12 @@
 	</tr><tr>
 		<th>Shell</th>
 		<th>3</th>
-		<th>987</th>
-		<th>136</th>
-		<th>37</th>
-		<th>814</th>
-		<th>91</th>
-		<th>35251</th>
+		<th>997</th>
+		<th>133</th>
+		<th>84</th>
+		<th>780</th>
+		<th>86</th>
+		<th>35707</th>
 	</tr><tr>
 		<th>JavaServer Pages</th>
 		<th>2</th>
@@ -350,16 +350,16 @@
 		<th>0</th>
 		<th>563</th>
 		<th>0</th>
-		<th>8263</th>
+		<th>8264</th>
 	</tr><tr>
 		<th>JSON</th>
 		<th>1</th>
-		<th>7162</th>
+		<th>7124</th>
 		<th>0</th>
 		<th>0</th>
-		<th>7162</th>
+		<th>7124</th>
 		<th>0</th>
-		<th>99807</th>
+		<th>99237</th>
 	</tr><tr>
 		<th>Korn Shell</th>
 		<th>1</th>
@@ -552,12 +552,12 @@
 	</tr></tbody>
 	<tfoot><tr>
 		<th>Total</th>
-		<th>162</th>
-		<th>24731</th>
-		<th>2830</th>
-		<th>1542</th>
-		<th>20359</th>
-		<th>2276</th>
-    	<th>1746225</th>
+		<th>165</th>
+		<th>24834</th>
+		<th>2843</th>
+		<th>1601</th>
+		<th>20390</th>
+		<th>2284</th>
+    	<th>1749940</th>
 	</tr></tfoot>
 	</table></body></html>
diff --git a/examples/issue214/ббббббббббббббббббббббббббббббббб.md b/examples/issue214/ббббббббббббббббббббббббббббббббб.md
diff --git a/examples/issue214/ѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬ.md b/examples/issue214/ѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬѬ.md
diff --git a/examples/issue214/中文中文中文中文中文中文中文中文中文中文中文中文中文中文中文中文.md b/examples/issue214/中文中文中文中文中文中文中文中文中文中文中文中文中文中文中文中文.md
diff --git a/go.mod b/go.mod
@@ -6,9 +6,9 @@ require (
 	github.com/dbaggerman/cuba v0.3.2
 	github.com/inconshreveable/mousetrap v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.9
+	github.com/mattn/go-runewidth v0.0.9
 	github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
-	github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742 // indirect
 	github.com/spf13/cobra v0.0.3
 	github.com/spf13/pflag v1.0.3 // indirect
 	golang.org/x/text v0.3.0

diff --git a/go.sum b/go.sum
@@ -1,4 +1,5 @@
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dbaggerman/cuba v0.3.2 h1:6ZbQX3FNvkocR222YyoAIZ8wi4avrb7JcJkPvVI2AS4=
 github.com/dbaggerman/cuba v0.3.2/go.mod h1:t9Oo05XRZGcjaVqsA/gFeNAOm7DYZYNhI17unI5FlwY=
@@ -8,25 +9,28 @@ github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANyt
 github.com/json-iterator/go v1.1.9 h1:9yzud/Ht36ygwatGx56VwCZtlI/2AD15T1X2sjSuGns=
 github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
 github.com/karrick/godirwalk v1.10.12/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0LhBygSwrAsHA=
+github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
+github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
 github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g=
 github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742 h1:Esafd1046DLDQ0W1YjYsBW+p8U2u7vzgW2SQVmlNazg=
 github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
-github.com/monochromegane/go-gitignore v0.0.0-20160105113617-38717d0a108c h1:RRUev95N3Gq4Aog4avFzXJA2V8fgXmND+cvcH7KLMyk=
-github.com/monochromegane/go-gitignore v0.0.0-20160105113617-38717d0a108c/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4=
 github.com/pkg/profile v1.3.0/go.mod h1:hJw3o1OdXxsrSjjVksARp5W95eeEaEfptyVZyv6JUPA=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/spf13/cobra v0.0.3 h1:ZlrZ4XsMRm04Fr5pSFxBgfND2EBVa1nLpiy1stUsX/8=
 github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
 github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
diff --git a/languages.json b/languages.json
@@ -1040,46 +1040,6 @@
     ],
     "quotes": []
   },
-  "Cargo Lock": {
-    "complexitychecks": [
-      "for ",
-      "for(",
-      "if ",
-      "if(",
-      "switch ",
-      "while ",
-      "else ",
-      "|| ",
-      "&& ",
-      "!= ",
-      "== "
-    ],
-    "extensions": [
-      "cargo.lock"
-    ],
-    "line_comment": [
-      "#"
-    ],
-    "multi_line": [],
-    "quotes": [
-      {
-        "end": "\\\"",
-        "start": "\\\""
-      },
-      {
-        "end": "'",
-        "start": "'"
-      },
-      {
-        "end": "\\\"\\\"\\\"",
-        "start": "\\\"\\\"\\\""
-      },
-      {
-        "end": "'''",
-        "start": "'''"
-      }
-    ]
-  },
   "Cassius": {
     "complexitychecks": [
       "for ",
@@ -2463,7 +2423,9 @@
       "{{ with"
     ],
     "extensions": [
-      "tmpl"
+      "tmpl",
+      "gohtml",
+      "gotxt"
     ],
     "line_comment": [],
     "multi_line": [

diff --git a/main.go b/main.go
@@ -93,7 +93,7 @@ func main() {
 		"format",
 		"f",
 		"tabular",
-		"set output format [tabular, wide, json, csv, cloc-yaml, html, html-table, sql, sql-insert]",
+		"set output format [tabular, wide, json, csv, csv-stream, cloc-yaml, html, html-table, sql, sql-insert]",
 	)
 	flags.StringSliceVarP(
 		&processor.AllowListExtensions,

diff --git a/processor/constants.go b/processor/constants.go