All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
13.0.0-alpha.1 - 2024-03-04
- fixed language names not showing when in Light mode (light background (#1048)
- Create release-plz.yaml
- Update mean_bean_ci.yml
- Fix LD Script language data (#1028)
- Fix language data example in CONTRIBUTING.md (#1029)
- Update dependencies
- Add widget install instructions
- Update mean_bean_ci.yml
- Dockerize tokei (#930)
- Ignore format commits for
languages.json(#1013) - Upgrade GitHub Actions (#955)
- add --languages ouput formatter (#1007)
- Add Nuget Config, Bazel and EdgeQL Support, Fix Output Formatter (#999)
- show nushell in the readme (#991)
- Add support for Redscript (#994)
- Add support for jq (#965)
- Add support for Astro (#966)
- Use XDG conventions on macOS too (#989)
- Add JSON5 support for languages.json (#986)
- Delete Smalltalk.cs.st (#990)
- Add support for smalltalk (#839)
- Disable *-android
- Add HiCAD to languages.json (#985)
- Add Nushell to languages.json (#982)
Tokei is a fast and accurate code analysis CLI tool and library, allowing you to easily and quickly see how many blank lines, comments, and lines of code are in your codebase. All releases and work on Tokei and tokei.rs (the free companion badge service) are funded by the community through GitHub Sponsors.
You can always download the latest version of tokei through GitHub Releases or Cargo. Tokei is also available through other package managers, though they may not always contain the latest release.
cargo install tokei
- Added
-n/--num-format=[commas, dots, plain, underscores]for adding separator formatting for numbers. - The total is now included in output formats such as JSON.
--no-ignorenow implies other ignore flags.- Added
--no-ignore-dotflag to ignore files such as.ignore. - Added single line comments to F*
- Updated various dependencies.
Tokei 12 comes with some of the biggest user facing changes since 1.0, now in the latest version tokei will now analyse and count multiple languages embedded in your source code as well as adding support for Jupyter Notebooks. Now for the first time is able to handle and display different languages contained in a single source file. This currently available for a limited set of languages, with plans to add more support for more in the future. The currently supported languages are;
Tokei will now analyse and report the source code contained in <script>,
<style>, and <template> tags in HTML and other similar languages. Tokei will
read the value of thetype attribute from the <script> tag and detects the
appropriate language based on its mime type or JavaScript if not present. Tokei
will do the same for <style> and <template> except reading the lang
attribute instead of type and defaulting to CSS and HTML each respectively.
Tokei will now read Jupyter Notebook files (.ipynb) and will read the source
code and markdown from Jupyter's JSON and output the analysed result.
Tokei will now detect any code blocks marked with specified source language and count each as their respective languages or as Markdown if not present or not found. Now you can easily see how many code examples are included in your documentation.
Tokei will now detect blocks of rustdoc documentation (e.g. //////!) and
parse them as markdown.
Tokei is now also capable of handling "verbatim" strings, which are strings that
do not accept escape sequences like \. Thanks to @NickHackman for providing
the implementation! This is initially supported for C++, C#, F#, and Rust.
To be able to show these new features, tokei's output has been changed to look like below. For brevity the CLI only displays one level deep in each language, however the library's parser is fully recursive and you can get access to the complete report using the library or by outputting the JSON format.
===============================================================================
Language Files Lines Code Comments Blanks
===============================================================================
BASH 4 49 30 10 9
JSON 1 1332 1332 0 0
Shell 1 49 38 1 10
TOML 2 77 64 4 9
-------------------------------------------------------------------------------
Markdown 5 1230 0 965 265
|- JSON 1 41 41 0 0
|- Rust 2 53 42 6 5
|- Shell 1 22 18 0 4
(Total) 1346 101 971 274
-------------------------------------------------------------------------------
Rust 19 3349 2782 116 451
|- Markdown 12 351 5 295 51
(Total) 3700 2787 411 502
===============================================================================
Total 32 6553 4352 1397 804
===============================================================================
This feature is not just limited to the default output of tokei. You can see it
broken down by each file with the --files option.
===============================================================================
Language Files Lines Code Comments Blanks
===============================================================================
Markdown 5 1230 0 965 265
|- JSON 1 41 41 0 0
|- Rust 2 53 42 6 5
|- Shell 1 22 18 0 4
(Total) 1346 101 971 274
-------------------------------------------------------------------------------
./CODE_OF_CONDUCT.md 46 0 28 18
./CHANGELOG.md 570 0 434 136
-- ./markdown.md --------------------------------------------------------------
|- Markdown 4 0 3 1
|- Rust 6 4 1 1
|- (Total) 10 4 4 2
-- ./README.md ----------------------------------------------------------------
|- Markdown 498 0 421 77
|- Shell 22 18 0 4
|- (Total) 520 18 421 81
-- ./CONTRIBUTING.md ----------------------------------------------------------
|- Markdown 112 0 79 33
|- JSON 41 41 0 0
|- Rust 46 38 4 4
|- (Total) 200 79 84 37
===============================================================================
Total 5 1346 101 971 274
===============================================================================
- The JSON Output and format of
Languageshas changed. - The JSON feature has been removed and is now included by default.
Statshas been split intoReportandCodeStatsto better represent the separation between analysing a file versus a blob of code.
- @alexmaco Added shebang and env detection for Crystal.
- @NickHackman Updated both Vue and HTML to count CSS & JS comments as comments.
- @XAMPPRocky renamed Perl6's display name to Rakudo.
- @dbackeus Added
erbextension for Ruby HTML. - @kobataiwan Tokei will now check for a configuration file in your home directory as well as your current and configuration directory.
- @dependabot Updated dependencies
Added Languages
- @alexmaco Dhall
- @NickHackman Svelte
- @athas Futhark
- @morphy2k Gohtml
- @LucasMW Headache
- @rosasynstylae Tsx
- @XAMPPRocky OpenType Feature Files
Added Languages
-
@rubdos Arduino
-
@LuqueDaniel Pan
-
@itkovian Ren'Py
-
Added
LanguageType::shebangs,LanguageType::from_file_extension, andLanguageType::from_shebang. (@solanav)
Added languages
- @bwidawsk GNU Assembly, GDB Script
- @isker Dust, Apache Velocity
- @andreblanke FreeMarker
Thanks to some major internal refactoring, Tokei has received significant performance improvements, and is now one of the fastest code counters across any size of codebase. With Tokei 11 showing up to 40–60% faster results than tokei's previous version. To showcase the improvements I've highlighted benchmarks of counting five differently sized codebases. Redis (~220k lines), Rust (~16M lines), and the Unreal Engine (~37.5M lines). In every one of these benchmarks Tokei 11 performed the best by a noticeable margin.
All benchmarks were done on a 15-inch MacBook Pro, with a 2.7GHz Intel Core i7 processor and 16GB 2133 MHz LPDDR3 RAM running macOS Catalina 10.15.3. Your mileage may vary, All benchmarks were done using hyperfine, using default settings for all programs.
Note This benchmark is not accurate due to tokei and loc both taking
less than 5ms to complete, there is a high degree of error between the times and
should mostly be considered equivalent. However it is included because it is
notable that scc takes nearly 3x as long to complete on smaller codebases
(~5k lines).
- Added
pywextension to Python. - Updated dependencies
- Fixed
.tokeignorealways working even when--no-ignoreis present. - Updated dependencies
Added languages
- @erikaxel Gherkin (Cucumber)
- Added
cjsxextension to CoffeeScript. - Tokei will now recognise files with
#!/usr/bin/env rubyas Ruby. - Updated dependencies.
- Tokei now uses
crossbeamchannels overstd::mpsc, which should have a noticeable performance improvement on large repos. - Improved documentation for
libtokei.
Added languages
- @lzybkr PowerShell
- @turbo MoonScript
- @dtolnay Thrift
- @Tranzystorek FlatBuffers
- @NieDzejkob Emojicode
- @DanteFalzone0 HolyC
- @sci4me Odin
- @fkarg Rusty Object Notation (RON)
- Fixed minor parsing bugs.
- Width is now limited to 80 unless you use the
--filesflag. - Added the
mjsextension to JavaScript. - Added the
tppextension to C++. - You can now disable Tokei's git ignore detection, similar to ripgrep. See
--helpfor options. - You can now add a
.tokeignorefile to your project to specify file paths for tokei to always ignore. This file uses the same syntax as.gitignore. - Improved Pascal representation
Added languages
- @hobofan solidity
- @stefanmaric GraphQL
- @jhpratt PostCSS
- @evitalis RPM
- @alexmaco Pony
- @yjhmelody WASM, LLVM, Pest
- @XAMPPRocky ASN.1
- Tokei now has config files. You can now specify some commonly used arguments
in a
.tokeirc/tokei.toml. Namelycolumnsto set the default column output,typesto filter your count to just a single set of languages, andtreat_doc_strings_as_commentswhich is a new option that allows you to specify whether to treat doc strings such as"""in Python as comments or code. The config files can be specified in two places, the current directory tokei is running in and your system configuration directory. The priority of options is as followsCLI > <current_directory> > <configuration_directory>. - Tokei is now available on Conda.
- Tokei's README has been translated to chinese.
LanguageTypenow implementsHash.- Tokei now batches it's console output, this should result in a small performance boost.
- There is now a
--columnsargument for manually setting tokei's output width. - The
--sortargument is now case-insensitive. - Tokei will now mark languages who's files failed to parse correctly as potentially inaccurate.
- Due to a bug in trust-ci
x86_64-unknown-netbsdversions are will not be available in GitHub releases. (You will still be able to install from source.) - Due to toml-rs's lacking enum support the TOML output option has been disabled.
Added languages
- @t-richards Liquid
- @diaphore Added the
.glslextension to GLSL. - @ahmedelgabri Twig
- @pmoura Logtalk
- @alekratz Perl, Not Quite Perl
- @XAMPPRocky Automake, .NET Resource, HLSL, INI, Unreal Plugin, Unreal Project, Unreal Shader, Unreal Shader Header, Unreal Markdown, Visual Basic, Visual Studio Solution, Visual Studio Project, Xcode Config,
- @TheMrNomis SWIG
- @xnorme Added the
.vhdlextension to VHDL
- A language's comments, and quotes are now available through the
LanguageTypeenum. - You can filter by language using the
-t/--typeoption. e.g.tokei -t "Rust,C"will print only Rust and C files. - Tokei now understands terminal width and will expand to fit it. (Thanks to @Veykril)
- Added comparison document to compare Tokei to other code counters.
- Updated dependencies
Added languages
- @BrandonBoone VB6, VBScript, XSLT
- @ialpert BrightScript
- @PJB3005 Dream Maker
- @schmee edn
Made various optimisations, up to 65% faster in some cases.
Added languages
- @DenialAdams Added Forsyth-Edwards-Notation (FEN)
- @DjebbZ Added ClojureC
- @grimm26 Added HCL/Terraform
- Updated dependencies.
- Changed how compilied serialization formats are handled.
- Fixed minor parser inaccuracies.
- Tokei should now recognise more python files from their shebang.
Added languages
- @ignatenko Added Meson
- @sprang Added Scheme
- @fengcms Added Vue
- @mark.knol Added Haxe
- @rleungx Added ABAP, COBOL, and Groovy
- @tiehuis Added Zig
- @murielsilveira Added Mint
- @notramo Added Elvish Shell and Kakoune
- @aatxe Added Racket
- @kamilchm Added ReasonML
- @cyplp Added XSL
- Updated dependencies
- Fixed parsing corner cases
- Changed storage of comments and quotes from
Vecto static slices. - Added tracing for debugging single files. Not recommended for use on multiple file
- Updated
log
- Fixed inaccuracies relating to the end comment being smaller than start comment.
Added languages
- @mattico Added Xaml
- @weakish Added Ceylon
- @theduke Added tsx extension to typescript
- @vmchale Added Hamlet, Cassius, Lucius, Cabal, Nix, Happy, Alex, and Madlang
- @notramo Added Crystal
- Now can recognise file languages based on their filename.
Added Languages:
- @kazimuth CMake, Dockerfile, Rakefile, Scons
- Multiple exclude flags now allowed.
Added Languages:
- @seiks Added Fish Shell
- @XAMPPRocky Added Module-Definition
- @tbu- Added Vala
- Reworked internals
- Now uses serdederive(_and thusly requires rust v1.15)
- Now has better file based testing
Added languages:
- @tuncer Added Ur/Web
- @svisser Added PureScript
- @tjodden Add some common extensions for HTML, C++ and Makefile
- @xd009642 Added VHDL
- Optimised internals
Added languages:
- @GungnirInd Added GDScript
- @tuncer Differentiate between sh and Bash, Added Cogent, F*, F#
- @pthariensflame Added Agda
- Added Regex based hueristics so more expensive multi line handling isn't used if there are no multi line comments in the file.
- Now uses the
ignorecrate for getting files. Which now also makes determining language from path/file parallelised - File counting used to only be parallelised per language, now it is also parallelised per file per language.
- Updated homepage, and documentation links
- @rmbreak Tokei will now not add directories with
foo.barlike syntax to a language. - @Michael-F-Bryan tokei will now exit gracefully when a feature is missing instead of panicking
Added languages:
- @hauleth Added Elixir support
- Simplified language definitions, now consolidated into a single JSON file.
- Fixed regression where lines and files weren't sorted.
- @llogiq : made clippy fixes
- @lligo : Added long verbose name
Added languages:
- @little-dude : Tcl(tcl)
- @svenstaro : GLSL(vert, tesc, tese, geom, frag, comp)
- @not-fl3 : Elm(elm)
Changes to existing languages:
- @xpayn : Added
pmextension to Perl.
- @lligo : Tokei no longer panics on non-character-boundary when printing file names.
- Fixed regression where no comment style files(json, markdown) weren't counted.
- Tokei can now handle files in different encodings.(using the encoding library)
- Tokei now prints errors instead of silently skipping them.
- Tokei can now print unused extensions using
-voption.
Added languages:
- Asp(asa, asp)
- Asp.NET(asax, ascx, asmx, aspx, master, sitemap, webinfo)
- Hex(hex)
- Intel Hex(ihex)
- ReStructuredText(rst)
- Razor(cshtml)
Changes to existing languages Thanks to @mwilli20 :
- Another Ada extension(pad)
- Assembly - Uses
' 'or" "and added another extension(asm) - Bash - Uses
' 'or" " - Batch - They don't use quotes for strings, added
:: - Cold Fusion - Uses
' 'or" " - D - Uses
" "or - Dart - Uses
" "or' 'or""" """or''' ''' - Forth - Uses
" "but new, doesn't have a preset - Fortrans - Use
" "or' ' - Idris - Uses
" "or""" """ - Julia - Uses
" "or""" """ - Kotlin - Uses
" "or""" """ - Lisp - Comments can be nested
- Moustache - Uses
" "or' ' - Nim - Uses
" "or""" """ - Pascal - Uses
' ' - Perl - Uses
" "or' ' - Php - Uses
" "or' ' - Python - Uses
" "or' 'or""" """or''' ''' - Ruby - Uses
" "or' ' - Sass - Uses
" "or' ' - Sql - Uses
' ' - Toml - Uses
" "or' 'or""" """or''' ''' - Typescript - Uses
" "or' 'or - Vimscript - Uses
" "or' ' - Yaml - Uses
" "or' ' - Zsh - Uses
" "or' ' - Clojure - Removed
# - Forth -
( Comment)style comments need a space after the opening paren - Haskell - Has nested comments
- Idris - Has nested comments
- Jai - Has nested block comments
- Julia - Has nested block comments
- Kotlin - Has nested block comments
- Pascal - Pascal should be multiline from
{or(*to}or*) - Perl - Perl5 and earlier for multiline comments need
=podto=cut. - Swift - Has nested block comments
-------------------------------------------------------------------------------
Language Files Lines Code Comments Blanks
-------------------------------------------------------------------------------
Rust 13 2413 1596 601 216
-------------------------------------------------------------------------------
|ib\language\languages.rs 693 420 197 76
|anguage\language_type.rs 500 386 102 12
.\src\main.rs 314 256 17 41
|lib\language\language.rs 356 166 166 24
.\src\lib\utils\fs.rs 129 107 9 13
|\lib\utils\multi_line.rs 149 89 39 21
.\src\lib\utils\macros.rs 59 50 3 6
.\src\lib\stats.rs 63 45 12 6
.\src\lib\lib.rs 76 25 47 4
.\src\lib\build.rs 31 23 0 8
.\src\lib\sort.rs 28 19 6 3
.\src\lib\language\mod.rs 11 6 3 2
.\src\lib\utils\mod.rs 4 4 0 0
-------------------------------------------------------------------------------
Markdown 4 492 492 0 0
-------------------------------------------------------------------------------
.\README.md 252 252 0 0
.\CHANGELOG.md 202 202 0 0
.\CONTRIBUTING.md 25 25 0 0
.\CONTRIBUTORS.md 13 13 0 0
-------------------------------------------------------------------------------
YAML 2 70 67 3 0
-------------------------------------------------------------------------------
.\cli.yml 53 50 3 0
.\.travis.yml 17 17 0 0
-------------------------------------------------------------------------------
TOML 1 80 65 0 15
-------------------------------------------------------------------------------
.\Cargo.toml 80 65 0 15
-------------------------------------------------------------------------------
Autoconf 1 9 7 1 1
-------------------------------------------------------------------------------
.\src\lib\lib.rs.in 9 7 1 1
-------------------------------------------------------------------------------
Total 21 3064 2227 605 232
-------------------------------------------------------------------------------
Tokei is now more precise, and shouldn't ever panic. Tokei now handles comments in quotes and more precise nested comments properly. Fixes #53
-------------------------------------------------------------------------------
Language Files Lines Code Comments Blanks
-------------------------------------------------------------------------------
Rust 13 2303 1487 594 222
-------------------------------------------------------------------------------
|ib\language\languages.rs 682 401 198 83
|anguage\language_type.rs 467 359 96 12
.\src\main.rs 302 243 17 42
|lib\language\language.rs 356 166 166 24
.\src\lib\utils\fs.rs 116 95 9 12
|\lib\utils\multi_line.rs 156 93 41 22
.\src\lib\stats.rs 54 36 12 6
.\src\lib\build.rs 31 23 0 8
.\src\lib\lib.rs 69 22 43 4
.\src\lib\utils\macros.rs 27 20 3 4
.\src\lib\sort.rs 28 19 6 3
.\src\lib\language\mod.rs 11 6 3 2
.\src\lib\utils\mod.rs 4 4 0 0
-------------------------------------------------------------------------------
YAML 2 68 65 3 0
-------------------------------------------------------------------------------
.\cli.yml 49 46 3 0
.\.travis.yml 19 19 0 0
-------------------------------------------------------------------------------
TOML 1 71 58 0 13
-------------------------------------------------------------------------------
.\Cargo.toml 71 58 0 13
-------------------------------------------------------------------------------
Autoconf 1 9 7 1 1
-------------------------------------------------------------------------------
.\src\lib\lib.rs.in 9 7 1 1
-------------------------------------------------------------------------------
Total 17 2451 1617 598 236
-------------------------------------------------------------------------------
Tokei is now ~40% faster.
Added languages
- Ada
- Forth
Tokei now has a minimal version without serde for faster compilation.
Updated various dependencies.
Internal dependencies removed.
- CBOR is not supported till it supports
serde 0.8
Added languages
- Handlebars
Tokei is now available as a library.
Tokei now has a lot more tests.
Tokei now supports TOML
Fixed #41
Fixed #44
Fixed #45
Tokei, can now output results in various formats(cbor, json, yaml)
Conversely tokei can now take in results in those formats, and add them to the current run.
Premilarily support for nested comments(currently only supported for rust)
Change in the output format PR #35
Moved .sc from Lisp to Scala.
Internals changed to allow for multiple multi line comment formats.
Added languages:
- Isabelle
Major rewrite, now parallelized. Can now support sorting files. Added a progress message for when it is counting files. Fixed #29
Added languages:
- Coq
- Erlang
- Kotlin
- Idris
- Nim
- Oz
- Prolog
- Qcl
- Scala
- Unreal Script
- Wolfram
Added file counting.
Added Shebang support.
Added languages:
- Assembly
- LD Scripts
- Device Trees
- Makefiles
- Plain Text
- C Shell
Changed the formatting so tokei looks nice for consoles of 80 column width.
Changed from handmade recursive file opening to walkdir