Skip to content

Commit cc2dd5b

Browse files
alecthomasalecthomas
committed
Version 2 of Chroma
This cleans up the API in general, removing a bunch of deprecated stuff, cleaning up circular imports, etc. But the biggest change is switching to an optional XML format for the regex lexer. Having lexers defined only in Go is not ideal for a couple of reasons. Firstly, it impedes a significant portion of contributors who use Chroma in Hugo, but don't know Go. Secondly, it bloats the binary size of any project that imports Chroma. Why XML? YAML is an abomination and JSON is not human editable. XML also compresses very well (eg. Go template lexer XML compresses from 3239 bytes to 718). Why a new syntax format? All major existing formats rely on the Oniguruma regex engine, which is extremely complex and for which there is no Go port. Why not earlier? Prior to the existence of fs.FS this was not a viable option. Benchmarks: $ hyperfine --warmup 3 \ './chroma.master --version' \ './chroma.xml-pre-opt --version' \ './chroma.xml --version' Benchmark 1: ./chroma.master --version Time (mean ± σ): 5.3 ms ± 0.5 ms [User: 3.6 ms, System: 1.4 ms] Range (min … max): 4.2 ms … 6.6 ms 233 runs Benchmark 2: ./chroma.xml-pre-opt --version Time (mean ± σ): 50.6 ms ± 0.5 ms [User: 52.4 ms, System: 3.6 ms] Range (min … max): 49.2 ms … 51.5 ms 51 runs Benchmark 3: ./chroma.xml --version Time (mean ± σ): 6.9 ms ± 1.1 ms [User: 5.1 ms, System: 1.5 ms] Range (min … max): 5.7 ms … 19.9 ms 196 runs Summary './chroma.master --version' ran 1.30 ± 0.23 times faster than './chroma.xml --version' 9.56 ± 0.83 times faster than './chroma.xml-pre-opt --version' A slight increase in init time, but I think this is okay given the increase in flexibility. And binary size difference: $ du -h lexers.test* $ du -sh chroma* 951371ms 8.8M chroma.master 7.8M chroma.xml 7.8M chroma.xml-pre-opt Benchmarks: $ hyperfine --warmup 3 \ './chroma.master --version' \ './chroma.xml-pre-opt --version' \ './chroma.xml --version' Benchmark 1: ./chroma.master --version Time (mean ± σ): 5.3 ms ± 0.5 ms [User: 3.6 ms, System: 1.4 ms] Range (min … max): 4.2 ms … 6.6 ms 233 runs Benchmark 2: ./chroma.xml-pre-opt --version Time (mean ± σ): 50.6 ms ± 0.5 ms [User: 52.4 ms, System: 3.6 ms] Range (min … max): 49.2 ms … 51.5 ms 51 runs Benchmark 3: ./chroma.xml --version Time (mean ± σ): 6.9 ms ± 1.1 ms [User: 5.1 ms, System: 1.5 ms] Range (min … max): 5.7 ms … 19.9 ms 196 runs Summary './chroma.master --version' ran 1.30 ± 0.23 times faster than './chroma.xml --version' 9.56 ± 0.83 times faster than './chroma.xml-pre-opt --version' Incompatible changes: - (*RegexLexer).SetAnalyser: changed from func(func(text string) float32) *RegexLexer to func(func(text string) float32) Lexer - (*TokenType).UnmarshalJSON: removed - Lexer.AnalyseText: added - Lexer.SetAnalyser: added - Lexer.SetRegistry: added - MustNewLazyLexer: removed - MustNewLexer: changed from func(*Config, Rules) *RegexLexer to func(*Config, func() Rules) *RegexLexer - Mutators: changed from func(...Mutator) MutatorFunc to func(...Mutator) Mutator - NewLazyLexer: removed - NewLexer: changed from func(*Config, Rules) (*RegexLexer, error) to func(*Config, func() Rules) (*RegexLexer, error) - Pop: changed from func(int) MutatorFunc to func(int) Mutator - Push: changed from func(...string) MutatorFunc to func(...string) Mutator - TokenType.MarshalJSON: removed - Using: changed from func(Lexer) Emitter to func(string) Emitter - UsingByGroup: changed from func(func(string) Lexer, int, int, ...Emitter) Emitter to func(int, int, ...Emitter) Emitter
1 parent d491f1b commit cc2dd5b

File tree

490 files changed

+32816
-14470
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

490 files changed

+32816
-14470
lines changed

Diff for: ‎.github/workflows/release.yml

+3-1
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ jobs:
1212
with:
1313
fetch-depth: 0
1414
- run: ./bin/hermit env --raw >> $GITHUB_ENV
15-
- run: goreleaser release --rm-dist
15+
- run: find ./lexers -name '*.xml' | xargs gzip -9
16+
- run: git add lexers
17+
- run: goreleaser release --rm-dist --skip-validate
1618
env:
1719
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Diff for: ‎.golangci.yml

+4-2
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ linters:
3636
- ifshort
3737
- wrapcheck
3838
- stylecheck
39+
- thelper
3940

4041
linters-settings:
4142
govet:
@@ -48,8 +49,8 @@ linters-settings:
4849
min-len: 8
4950
min-occurrences: 3
5051
forbidigo:
51-
forbid:
52-
- (Must)?NewLexer
52+
#forbid:
53+
# - (Must)?NewLexer$
5354
exclude_godoc_examples: false
5455

5556

@@ -74,3 +75,4 @@ issues:
7475
- 'methods on the same type should have the same receiver name'
7576
- '_TokenType_name should be _TokenTypeName'
7677
- '`_TokenType_map` should be `_TokenTypeMap`'
78+
- 'rewrite if-else to switch statement'

Diff for: ‎_tools/css2style/main.go

+2-2
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,14 @@ import (
1010
"github.com/aymerick/douceur/parser"
1111
"gopkg.in/alecthomas/kingpin.v3-unstable"
1212

13-
"github.com/alecthomas/chroma"
13+
"github.com/alecthomas/chroma/v2"
1414
)
1515

1616
const (
1717
outputTemplate = `package styles
1818
1919
import (
20-
"github.com/alecthomas/chroma"
20+
"github.com/alecthomas/chroma/v2"
2121
)
2222
2323
// {{.Name}} style.

Diff for: ‎_tools/exercise/main.go

+3-3
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@ import (
55
"io/ioutil"
66
"os"
77

8-
"github.com/alecthomas/chroma/formatters"
9-
"github.com/alecthomas/chroma/lexers"
10-
"github.com/alecthomas/chroma/styles"
8+
"github.com/alecthomas/chroma/v2/formatters"
9+
"github.com/alecthomas/chroma/v2/lexers"
10+
"github.com/alecthomas/chroma/v2/styles"
1111
"gopkg.in/alecthomas/kingpin.v3-unstable"
1212
)
1313

Diff for: ‎_tools/pygments2chroma.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@
1515
package {{package}}
1616
1717
import (
18-
. "github.com/alecthomas/chroma" // nolint
19-
"github.com/alecthomas/chroma/lexers/internal"
18+
. "github.com/alecthomas/chroma/v2" // nolint
19+
"github.com/alecthomas/chroma/v2/lexers/internal"
2020
)
2121
2222
// {{upper_name}} lexer.

Diff for: ‎_tools/style.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
package styles
1111
1212
import (
13-
"github.com/alecthomas/chroma"
13+
"github.com/alecthomas/chroma/v2"
1414
)
1515
1616
// {{upper_name}} style.

Diff for: ‎bin/.hyperfine-1.12.0.pkg

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
hermit

Diff for: ‎bin/hyperfine

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
.hyperfine-1.12.0.pkg

Diff for: ‎cmd/chroma/go.mod

+5-5
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
1-
module github.com/alecthomas/chroma/cmd/chroma
1+
module github.com/alecthomas/chroma/v2/cmd/chroma
22

3-
go 1.16
3+
go 1.17
44

5-
replace github.com/alecthomas/chroma => ../../
5+
replace github.com/alecthomas/chroma/v2 => ../../
66

77
require (
8-
github.com/alecthomas/chroma v0.0.0-00010101000000-000000000000
8+
github.com/alecthomas/chroma/v2 v2.0.0-00010101000000-000000000000
99
github.com/alecthomas/kong v0.2.17
10-
github.com/mattn/go-colorable v0.1.10
10+
github.com/mattn/go-colorable v0.1.12
1111
github.com/mattn/go-isatty v0.0.14
1212
)

Diff for: ‎cmd/chroma/go.sum

+6-8
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,14 @@
11
github.com/alecthomas/kong v0.2.17 h1:URDISCI96MIgcIlQyoCAlhOmrSw6pZScBNkctg8r0W0=
22
github.com/alecthomas/kong v0.2.17/go.mod h1:ka3VZ8GZNPXv9Ov+j4YNLkI8mTuhXyr/0ktSlqIydQQ=
3-
github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964 h1:y5HC9v93H5EPKqaS1UYVg1uYah5Xf51mBfIoWehClUQ=
4-
github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964/go.mod h1:Xd9hchkHSWYkEqJwUGisez3G1QY8Ryz0sdWrLPMGjLk=
3+
github.com/alecthomas/repr v0.0.0-20220113201626-b1b626ac65ae h1:zzGwJfFlFGD94CyyYwCJeSuD32Gj9GTaSi5y9hoVzdY=
4+
github.com/alecthomas/repr v0.0.0-20220113201626-b1b626ac65ae/go.mod h1:2kn6fqh/zIyPLmm3ugklbEi5hg5wS435eygvNfaDQL8=
55
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
66
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
77
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
88
github.com/dlclark/regexp2 v1.4.0 h1:F1rxgk7p4uKjwIQxBs9oAXe5CqrXlCduYEJvrF4u93E=
99
github.com/dlclark/regexp2 v1.4.0/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc=
10-
github.com/mattn/go-colorable v0.1.10 h1:KWqbp83oZ6YOEgIbNW3BM1Jbe2tz4jgmWA9FOuAF8bw=
11-
github.com/mattn/go-colorable v0.1.10/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
12-
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
10+
github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40=
11+
github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
1312
github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y=
1413
github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
1514
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@@ -19,10 +18,9 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
1918
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
2019
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
2120
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
22-
golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
23-
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
24-
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c h1:F1jZWGFhYfh0Ci55sIpILtKKK8p3i2/krTr0H1rg74I=
2521
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
22+
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6 h1:foEbQz/B0Oz6YIqu/69kfXPYeFQAuuMYFkjaqXzl5Wo=
23+
golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
2624
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
2725
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
2826
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=

0 commit comments

Comments
 (0)