Skip to content

Lexer number parsing refactor #11211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
straight-shoota merged 30 commits into from
Dec 10, 2021
Merged

Lexer number parsing refactor #11211

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
de76dd9
Lexer number parsing refactor
BlobCodes Sep 14, 2021
e4688c6
Fix numbers ending with e notation (ex. "12e")
BlobCodes Sep 15, 2021
83a00f8
Merge branch 'crystal-lang:master' into refactor/lexer-number-parsing
BlobCodes Oct 21, 2021
241a35d
Handle all "numeric literal without digits" errors in one place
BlobCodes Nov 29, 2021
e1db7da
Improve error message when integer has leading '0'
BlobCodes Nov 29, 2021
006e06b
Do not increment underscore counter on e-notation
BlobCodes Nov 29, 2021
a75e352
Fix specs
BlobCodes Nov 30, 2021
3b92b6d
Unify number suffix scanning
BlobCodes Nov 30, 2021
6b78d13
Change consecutive underscore error message
BlobCodes Dec 1, 2021
724585c
Refactor to apply rules in #11447 and fix bugs
BlobCodes Dec 1, 2021
14368ce
Merge branch 'master' into refactor/lexer-number-parsing
BlobCodes Dec 1, 2021
13af8b7
Make gen_check_int_fits_in_size less magic
BlobCodes Dec 1, 2021
57590fe
Allow underscore before e-notation in number
BlobCodes Dec 1, 2021
5d501d4
Improve code readability
BlobCodes Dec 1, 2021
66ce336
Revert "Improve code readability"
BlobCodes Dec 4, 2021
cfa567a
Remove duplicated statement
BlobCodes Dec 4, 2021
8fc9dd4
Improve code quality
BlobCodes Dec 4, 2021
e8799d5
Add more specs
BlobCodes Dec 4, 2021
b1d300a
Fix compiler
BlobCodes Dec 4, 2021
7ec5b1e
Add "unreachable" comment
BlobCodes Dec 4, 2021
ea610b4
Disallow number literals ending with e-notation
BlobCodes Dec 4, 2021
dabf5f3
Add spec for uppercase E-notation
BlobCodes Dec 4, 2021
470b367
Adapt "invalid suffix" error to compiler style
BlobCodes Dec 4, 2021
4ba109c
Add more invalid suffix specs
BlobCodes Dec 7, 2021
4779582
Use "unexpected '_' in number" error if underscore not actually trailing
BlobCodes Dec 7, 2021
794d887
Merge branch 'refactor/lexer-number-parsing' of https://github.com/Bl…
BlobCodes Dec 7, 2021
454c73b
Remove error_number_string, rename before_prefix_pos
BlobCodes Dec 8, 2021
5be0eaa
Fix hex literals with 17-19 digits possibly throwing a compiler error
BlobCodes Dec 8, 2021
f092dd6
Add specs for previous change
BlobCodes Dec 8, 2021
7ba5ddb
Remove suffix from previous spec
BlobCodes Dec 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 42 additions & 12 deletions spec/compiler/lexer/lexer_spec.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ describe "Lexer" do
["+1.0f32", "+1.0"], ["-1.0f32", "-1.0"], ["-0.0f32", "-0.0"], ["1_234.567_890_f32", "1234.567890"]]
it_lexes_f64 ["1.0", ["1.0hello", "1.0"], "+1.0", "-1.0", ["1_234.567_890", "1234.567890"]]
it_lexes_f32 [["1e+23_f32", "1e+23"], ["1.2e+23_f32", "1.2e+23"]]
it_lexes_f64 ["1e23", "1e-23", "1e+23", "1.2e+23", ["1e23f64", "1e23"], ["1.2e+23_f64", "1.2e+23"]]
it_lexes_f64 ["1e23", "1e-23", "1e+23", "1.2e+23", ["1e23f64", "1e23"], ["1.2e+23_f64", "1.2e+23"], "0e40", "2e01", ["2_e2", "2e2"], "1E40"]

it_lexes_number :i8, ["1i8", "1"]
it_lexes_number :i8, ["1_i8", "1"]
Expand Down Expand Up @@ -225,6 +225,7 @@ describe "Lexer" do
it_lexes_number :u64, ["10000000000000000000_u64", "10000000000000000000"]

it_lexes_i64 [["0x3fffffffffffffff", "4611686018427387903"]]
it_lexes_i64 [["-0x8000000000000000_i64", "-9223372036854775808"]]
it_lexes_i64 ["-9223372036854775808", "9223372036854775807"]
it_lexes_u64 [["0xffffffffffffffff", "18446744073709551615"]]

Expand Down Expand Up @@ -295,6 +296,8 @@ describe "Lexer" do
assert_syntax_error "118446744073709551616_u64", "118446744073709551616 doesn't fit in an UInt64"
assert_syntax_error "18446744073709551616_u64", "18446744073709551616 doesn't fit in an UInt64"
assert_syntax_error "-1_u64", "Invalid negative value -1 for UInt64"
assert_syntax_error "-0_u64", "Invalid negative value -0 for UInt64"
assert_syntax_error "-0u64", "Invalid negative value -0 for UInt64"

assert_syntax_error "18446744073709551616_i32", "18446744073709551616 doesn't fit in an Int32"
assert_syntax_error "9999999999999999999_i32", "9999999999999999999 doesn't fit in an Int32"
Expand All @@ -311,13 +314,21 @@ describe "Lexer" do
assert_syntax_error "18446744073709551616_u128", "18446744073709551616 doesn't fit in an UInt64. UInt128 literals that don't fit in an UInt64 are currently not supported"
assert_syntax_error "-1_u128", "Invalid negative value -1 for UInt128"

assert_syntax_error "1__1", "consecutive underscores in numbers aren't allowed"
assert_syntax_error "-3_", "trailing '_' in number"
assert_syntax_error "0b_10", "unexpected '_' in number"
assert_syntax_error "10e_10", "unexpected '_' in number"
assert_syntax_error "1_.1", "unexpected '_' in number"
assert_syntax_error "-0e_12", "unexpected '_' in number"

assert_syntax_error "0_12", "octal constants should be prefixed with 0o"
assert_syntax_error "0123", "octal constants should be prefixed with 0o"
assert_syntax_error "00", "octal constants should be prefixed with 0o"
assert_syntax_error "01_i64", "octal constants should be prefixed with 0o"

assert_syntax_error "0xFF_i8", "255 doesn't fit in an Int8"
assert_syntax_error "0o200_i8", "128 doesn't fit in an Int8"
assert_syntax_error "0b10000000_i8", "128 doesn't fit in an Int8"
assert_syntax_error "0xFF_i8", "0xFF doesn't fit in an Int8"
assert_syntax_error "0o200_i8", "0o200 doesn't fit in an Int8"
assert_syntax_error "0b10000000_i8", "0b10000000 doesn't fit in an Int8"

# 2**31 - 1
it_lexes_i32 [["0x7fffffff", "2147483647"], ["0o17777777777", "2147483647"], ["0b1111111111111111111111111111111", "2147483647"]]
Expand All @@ -326,28 +337,30 @@ describe "Lexer" do
it_lexes_i64 [["0xffffffff", "4294967295"], ["0o37777777777", "4294967295"], ["0b11111111111111111111111111111111", "4294967295"]]
# 2**32
it_lexes_i64 [["0x100000000", "4294967296"], ["0o40000000000", "4294967296"], ["0b100000000000000000000000000000000", "4294967296"]]
assert_syntax_error "0x100000000i32", "4294967296 doesn't fit in an Int32"
assert_syntax_error "0o40000000000i32", "4294967296 doesn't fit in an Int32"
assert_syntax_error "0b100000000000000000000000000000000i32", "4294967296 doesn't fit in an Int32"
assert_syntax_error "0x100000000i32", "0x100000000 doesn't fit in an Int32"
assert_syntax_error "0o40000000000i32", "0o40000000000 doesn't fit in an Int32"
assert_syntax_error "0b100000000000000000000000000000000i32", "0b100000000000000000000000000000000 doesn't fit in an Int32"
# 2**63 - 1
it_lexes_i64 [["0x7fffffffffffffff", "9223372036854775807"], ["0o777777777777777777777", "9223372036854775807"], ["0b111111111111111111111111111111111111111111111111111111111111111", "9223372036854775807"]]
# 2**63
it_lexes_u64 [["0x8000000000000000", "9223372036854775808"], ["0o1000000000000000000000", "9223372036854775808"], ["0b1000000000000000000000000000000000000000000000000000000000000000", "9223372036854775808"]]
assert_syntax_error "0x8000000000000000i64", "9223372036854775808 doesn't fit in an Int64"
assert_syntax_error "0o1000000000000000000000i64", "9223372036854775808 doesn't fit in an Int64"
assert_syntax_error "0b1000000000000000000000000000000000000000000000000000000000000000i64", "9223372036854775808 doesn't fit in an Int64"
assert_syntax_error "0x8000000000000000i64", "0x8000000000000000 doesn't fit in an Int64"
assert_syntax_error "0o1000000000000000000000i64", "0o1000000000000000000000 doesn't fit in an Int64"
assert_syntax_error "0b1000000000000000000000000000000000000000000000000000000000000000i64", "0b1000000000000000000000000000000000000000000000000000000000000000 doesn't fit in an Int64"
# 2**64 - 1
it_lexes_u64 [["0xffff_ffff_ffff_ffff", "18446744073709551615"], ["0o177777_77777777_77777777", "18446744073709551615"], ["0b11111111_11111111_11111111_11111111_11111111_11111111_11111111_11111111", "18446744073709551615"]]
it_lexes_u64 [["0x00ffffffffffffffff", "18446744073709551615"], ["0o001777777777777777777777", "18446744073709551615"], ["0b001111111111111111111111111111111111111111111111111111111111111111", "18446744073709551615"]]
# 2**64
assert_syntax_error "0x10000_0000_0000_0000", "0x10000_0000_0000_0000 doesn't fit in an UInt64"
assert_syntax_error "0x10000_0000_0000_0000_u64", "0x10000_0000_0000_0000 doesn't fit in an UInt64"
assert_syntax_error "0xfffffffffffffffff_u64", "0xfffffffffffffffff doesn't fit in an UInt64"
assert_syntax_error "0o200000_00000000_00000000", "0o200000_00000000_00000000 doesn't fit in an UInt64"
assert_syntax_error "0b100000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000", "0b100000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000 doesn't fit in an UInt64"
# Very large
assert_syntax_error "0x1afafafafafafafafafafaf", "0x1afafafafafafafafafafaf doesn't fit in an UInt64"
assert_syntax_error "0x1afafafafafafafafafafafi32", "0x1afafafafafafafafafafafi32 doesn't fit in an Int32"
assert_syntax_error "0x1afafafafafafafafafafafi32", "0x1afafafafafafafafafafaf doesn't fit in an Int32"
assert_syntax_error "0o1234567123456712345671234567", "0o1234567123456712345671234567 doesn't fit in an UInt64"
assert_syntax_error "0o12345671234567_12345671234567_i8", "0o12345671234567_12345671234567_i8 doesn't fit in an Int8"
assert_syntax_error "0o12345671234567_12345671234567_i8", "0o12345671234567_12345671234567 doesn't fit in an Int8"
assert_syntax_error "0b100000000000000000000000000000000000000000000000000000000000000000", "0b100000000000000000000000000000000000000000000000000000000000000000 doesn't fit in an UInt64"

it_lexes_i64 [["0o700000000000000000000", "8070450532247928832"]]
Expand All @@ -356,14 +369,31 @@ describe "Lexer" do
assert_syntax_error "4f33", "invalid float suffix"
assert_syntax_error "4f65", "invalid float suffix"
assert_syntax_error "4f22", "invalid float suffix"
assert_syntax_error "4i33", "invalid int suffix"
assert_syntax_error "4i65", "invalid int suffix"
assert_syntax_error "4i22", "invalid int suffix"
assert_syntax_error "4i3", "invalid int suffix"
assert_syntax_error "4i12", "invalid int suffix"
assert_syntax_error "4u33", "invalid uint suffix"
assert_syntax_error "4u65", "invalid uint suffix"
assert_syntax_error "4u22", "invalid uint suffix"
assert_syntax_error "4u3", "invalid uint suffix"
assert_syntax_error "4u12", "invalid uint suffix"
# Tests for #8782
assert_syntax_error "4F32", %(unexpected token: "F32")
assert_syntax_error "4F64", %(unexpected token: "F64")
assert_syntax_error "0F32", %(unexpected token: "F32")

assert_syntax_error "4.0_u32", "Invalid suffix u32 for decimal number"
assert_syntax_error "2e8i8", "Invalid suffix i8 for decimal number"

assert_syntax_error ".42", ".1 style number literal is not supported, put 0 before dot"
assert_syntax_error "-.42", ".1 style number literal is not supported, put 0 before dot"

assert_syntax_error "2e", "unexpected token: \"e\""
assert_syntax_error "2ef32", "unexpected token: \"ef32\""
assert_syntax_error "2e+_2", "unexpected '_' in number"

it "lexes not instance var" do
lexer = Lexer.new "!@foo"
token = lexer.next_token
Expand Down
Loading