Skip to content

New unreachable token value warning #2066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
parrt merged 10 commits into from
Dec 6, 2017
Merged

New unreachable token value warning #2066

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
da57241
Added checkForUnreachableTokens method, added new TOKEN_UNREACHABLE e…
KvanTTT May 14, 2017
585325f
Added testUnreachableTokens unit-test.
KvanTTT May 14, 2017
328cdf3
Fixed TOKEN_UNREACHABLE warning checks.
KvanTTT Oct 22, 2017
9324d07
Extend and fix unit tests.
KvanTTT Oct 22, 2017
58349e5
} on line by themselves.
KvanTTT Oct 28, 2017
f4c4e8b
Added comments to checkForUnreachableTokens and getSingleTokenValues …
KvanTTT Oct 28, 2017
64b2ced
Added check for fragment rules.
KvanTTT Oct 28, 2017
5b11aed
Restored formatting.
KvanTTT Nov 25, 2017
763eccc
Added tests with escaped characters.
KvanTTT Nov 25, 2017
07a0a80
Yes another formatting fixes.
KvanTTT Nov 25, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions tool-testsuite/test/org/antlr/v4/test/tool/TestSymbolIssues.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -401,4 +401,39 @@ public void testLabelsForTokensWithMixedTypesLRWithoutLabels() {

testErrors(test, false);
}

@Test public void testUnreachableTokens() {
String[] test = {
"lexer grammar Test;\n" +
"TOKEN1: 'as' 'df' | 'qwer';\n" +
"TOKEN2: [0-9];\n" +
"TOKEN3: 'asdf';\n" +
"TOKEN4: 'q' 'w' 'e' 'r' | A;\n" +
"TOKEN5: 'aaaa';\n" +
"TOKEN6: 'asdf';\n" +
"TOKEN7: 'qwer'+;\n" +
"TOKEN8: 'a' 'b' | 'b' | 'a' 'b';\n" +
"fragment\n" +
"TOKEN9: 'asdf' | 'qwer' | 'qwer';\n" +
Copy link
Member

@sharwell sharwell Nov 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Should include tokens with characters that need escapes, to make sure the values are properly escaped in the error messages.

Copy link
Member Author

@KvanTTT KvanTTT Nov 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

"TOKEN10: '\\r\\n' | '\\r\\n';\n" +
"TOKEN11: '\\r\\n';\n" +
"\n" +
"mode MODE1;\n" +
"TOKEN12: 'asdf';\n" +
"\n" +
"fragment A: 'A';",

"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:4:0: One of the token TOKEN3 values unreachable. asdf is always overlapped by token TOKEN1\n" +
"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:5:0: One of the token TOKEN4 values unreachable. qwer is always overlapped by token TOKEN1\n" +
"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:7:0: One of the token TOKEN6 values unreachable. asdf is always overlapped by token TOKEN1\n" +
"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:7:0: One of the token TOKEN6 values unreachable. asdf is always overlapped by token TOKEN3\n" +
"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:9:0: One of the token TOKEN8 values unreachable. ab is always overlapped by token TOKEN8\n" +
"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:11:0: One of the token TOKEN9 values unreachable. qwer is always overlapped by token TOKEN9\n" +
"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:12:0: One of the token TOKEN10 values unreachable. \\r\\n is always overlapped by token TOKEN10\n" +
"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:13:0: One of the token TOKEN11 values unreachable. \\r\\n is always overlapped by token TOKEN10\n" +
"warning(" + ErrorType.TOKEN_UNREACHABLE.code + "): Test.g4:13:0: One of the token TOKEN11 values unreachable. \\r\\n is always overlapped by token TOKEN10\n"
};

testErrors(test, false);
}
}
16 changes: 8 additions & 8 deletions tool-testsuite/test/org/antlr/v4/test/tool/TestToolSyntaxErrors.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,11 +269,11 @@ public void AllErrorCodesDistinct() {
"grammar A;\n" +
"tokens{Foo}\n" +
"b : Foo ;\n" +
"X : 'foo' -> popmode;\n" + // "meant" to use -> popMode
"Y : 'foo' -> token(Foo);", // "meant" to use -> type(Foo)
"X : 'foo1' -> popmode;\n" + // "meant" to use -> popMode
"Y : 'foo2' -> token(Foo);", // "meant" to use -> type(Foo)

"error(" + ErrorType.INVALID_LEXER_COMMAND.code + "): A.g4:4:13: lexer command popmode does not exist or is not supported by the current target\n" +
"error(" + ErrorType.INVALID_LEXER_COMMAND.code + "): A.g4:5:13: lexer command token does not exist or is not supported by the current target\n"
"error(" + ErrorType.INVALID_LEXER_COMMAND.code + "): A.g4:4:14: lexer command popmode does not exist or is not supported by the current target\n" +
"error(" + ErrorType.INVALID_LEXER_COMMAND.code + "): A.g4:5:14: lexer command token does not exist or is not supported by the current target\n"
};
super.testErrors(pair, true);
}
Expand All @@ -283,11 +283,11 @@ public void AllErrorCodesDistinct() {
"grammar A;\n" +
"tokens{Foo}\n" +
"b : Foo ;\n" +
"X : 'foo' -> popMode(Foo);\n" + // "meant" to use -> popMode
"Y : 'foo' -> type;", // "meant" to use -> type(Foo)
"X : 'foo1' -> popMode(Foo);\n" + // "meant" to use -> popMode
"Y : 'foo2' -> type;", // "meant" to use -> type(Foo)

"error(" + ErrorType.UNWANTED_LEXER_COMMAND_ARGUMENT.code + "): A.g4:4:13: lexer command popMode does not take any arguments\n" +
"error(" + ErrorType.MISSING_LEXER_COMMAND_ARGUMENT.code + "): A.g4:5:13: missing argument for lexer command type\n"
"error(" + ErrorType.UNWANTED_LEXER_COMMAND_ARGUMENT.code + "): A.g4:4:14: lexer command popMode does not take any arguments\n" +
"error(" + ErrorType.MISSING_LEXER_COMMAND_ARGUMENT.code + "): A.g4:5:14: missing argument for lexer command type\n"
};
super.testErrors(pair, true);
}
Expand Down
1 change: 1 addition & 0 deletions tool/src/org/antlr/v4/semantics/SemanticPipeline.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ public void process() {
}

symcheck.checkForModeConflicts(g);
symcheck.checkForUnreachableTokens(g);

assignChannelTypes(g, collector.channelDefs);

Expand Down
Loading