Parallel lock free testing, remove potential deadlocks, cache static data, go to descriptor via test by KvanTTT · Pull Request #3752 · antlr/antlr4

KvanTTT · 2022-06-19T17:56:22Z

@parrt I've fixed Java and C# runtimes. If it's ok, I'll unify other runtimes as well (where multithreading is actual)

Now Java runtime tests are superfast since they are inprocess (30s instead of 1.5 minutes)!

Also, now it's very easy to navigate to test data via tests in IntelliJ! See the video:

idea64_dBV9JpR0fg.mp4

Parallel runtime tests execution, fixes Parallel runtime tests execution #3370
Remove potential deadlocks in Java, fixes Remove potential deadlocks in Java #3756
Cache static data in ANTLR tool for performance, fixes Cache static data in ANTLR tool for performance (don't load every time from disk) #3757

runtime-testsuite/resources/descriptors/ParserExec/ListLabelForClosureContext.txt

runtime/Java/src/org/antlr/v4/runtime/atn/PredictionContext.java

tool/src/org/antlr/v4/Tool.java

parrt · 2022-06-20T20:39:11Z

A huge amount of work here. thank you!

KvanTTT · 2022-06-20T20:44:55Z

A huge amount of work here. thank you!

Thank you for your reviews and PR accepting :) I suggest changing other runtimes (moving EMPTY instances) to unify them with Java runtime (and probably to prevent potential deadlocks). I've complete it for Java and C#. It looks actual for C++, Swift., maybe Go.

parrt · 2022-06-20T20:49:26Z

Are we sure the others have deadlock issues? Not sure I have the energy to update. Maybe we can ping those target folks.

KvanTTT · 2022-06-20T20:54:41Z

Are we sure the others have deadlock issues? Not sure I have the energy to update. Maybe we can ping those target folks.

Not sure (because they can use another initialization mechanizm) but it's not fully excluded (C# is very similar to Java). But I also like the idea of synchonized API across runtimes. I can do that.

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

parrt · 2022-06-24T20:56:03Z

I had a random thought: Is any user code relying on EMPTY? I don't think so from a quick check but we want to be careful we don't break any user code as a result of this.

KvanTTT · 2022-06-25T17:21:34Z

I had a random thought: Is any user code relying on EMPTY? I don't think so from a quick check but we want to be careful we don't break any user code as a result of this.

Yes, we should be careful, but removing deadlocks is much more important than insignificant changing of API. And since we change Java code, code of other runtimes can be changed as well.

KvanTTT · 2022-06-25T17:29:10Z

I've almost completed everything, but I also want to update documentation.

Referencing subclass ParserRuleContext from superclass RuleContext initializer might lead to class loading deadlock Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

…odeCount Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Remove useless loadTemplates overloads Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

…ent navigation to tests data in IntelliJ Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

KvanTTT · 2022-06-25T22:25:25Z

I think now it's ready for review and merge.

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

tool/src/org/antlr/v4/codegen/Target.java

tool/src/org/antlr/v4/tool/ErrorManager.java

parrt · 2022-06-26T18:09:46Z

In the middle of the review but wanted to comment that this is very well structured...each commit does a logical unit of work. thanks! And amazing piece of refactoring overall 💯

runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTests.java

parrt · 2022-06-26T18:12:50Z

doc/antlr-project-testing.md


 All test are run in parallel both via maven and via IDE.

+In IntelliJ, it's very easy to go to source by right-click on any test and pressing `Jump to source` (F4).


"right-clicking"

runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers/RuntimeTestParser.java

runtime-testsuite/test/org/antlr/v4/test/runtime/FileUtils.java

doc/antlr-project-testing.md

KvanTTT · 2022-06-26T19:46:12Z

In the middle of the review but wanted to comment that this is very well structured...each commit does a logical unit of work. thanks! And amazing piece of refactoring overall 💯

I'd like to thank my @JetBrains team. They taught me to keep structured git history and write very conscious code.

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Fix misprint Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

parrt · 2022-06-26T21:36:02Z

An amazing PR if I ever saw one thanks!!! Merging...

See antlr/antlr4#3752 Commit 875911c2b2e6e50317e75ff2b5e80f4014a75954

KvanTTT force-pushed the parallel-lock-free-testing branch from 8b757f6 to b0f4531 Compare June 19, 2022 18:43

KvanTTT changed the title ~~Parallel lock free testing, remove potential deadlocks, cache static data~~ Parallel lock free testing, remove potential deadlocks, cache static data, go to descriptor via test Jun 20, 2022

KvanTTT force-pushed the parallel-lock-free-testing branch from a2c8d68 to 17f9e9d Compare June 20, 2022 15:05

parrt reviewed Jun 20, 2022

View reviewed changes

runtime-testsuite/resources/descriptors/ParserExec/ListLabelForClosureContext.txt Outdated Show resolved Hide resolved

parrt reviewed Jun 20, 2022

View reviewed changes

runtime/Java/src/org/antlr/v4/runtime/atn/PredictionContext.java Show resolved Hide resolved

parrt reviewed Jun 20, 2022

View reviewed changes

runtime/Java/src/org/antlr/v4/runtime/atn/PredictionContext.java Show resolved Hide resolved

parrt reviewed Jun 20, 2022

View reviewed changes

tool/src/org/antlr/v4/Tool.java Show resolved Hide resolved

KvanTTT added 3 commits June 21, 2022 15:35

Implement fast parallel lock-free in-process testing for Java

84c9780

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Add final modifier to all static fields where it's possible

e942e1a

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Remove doubtful and not used static fields

36b885c

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

KvanTTT force-pushed the parallel-lock-free-testing branch from 38e7869 to fb6fee8 Compare June 21, 2022 20:19

KvanTTT added 12 commits June 26, 2022 00:56

Fix potential deadlocks in Java runtime

4ccbb41

Referencing subclass ParserRuleContext from superclass RuleContext initializer might lead to class loading deadlock Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Unify C#, C++, Dart, Go, Python runtimes

dbce6b2

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Use AtomicInteger instead of int for static PredictionContext.globalN…

8861ac4

…odeCount Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Remove lock in Generator (now ANTLR tool is thread-safe)

3c30268

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Cache templates for codegen to static field, load only one time

7c5b3c2

Remove useless loadTemplates overloads Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Cache LeftRecursiveRules.stg to static field

f19cd67

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Cache ANTLR error message templates to static field

6681a7a

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Cache runtime test templates to static field

93970f4

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Clarify message of failed runtime tests

1b9d77c

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Pass testSourceUri to dynamicTest and to dynamicContainer for conveni…

99fab5c

…ent navigation to tests data in IntelliJ Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Extract prepareGrammars method

8e6e1ed

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Remove useless and outdated files

2e8e3a8

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Update pom.xml in runtime-testsuite

22cbf3c

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

KvanTTT force-pushed the parallel-lock-free-testing branch from fb6fee8 to ff655e3 Compare June 25, 2022 22:18

KvanTTT marked this pull request as ready for review June 25, 2022 22:25

KvanTTT added 2 commits June 26, 2022 01:51

Use runtimePath instead of targetClassesPath for runtimes' sources

d287f1c

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Update doc

f235d33

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

KvanTTT force-pushed the parallel-lock-free-testing branch from ff655e3 to f235d33 Compare June 25, 2022 22:51

parrt reviewed Jun 26, 2022

View reviewed changes

tool/src/org/antlr/v4/codegen/Target.java Show resolved Hide resolved

parrt reviewed Jun 26, 2022

View reviewed changes

tool/src/org/antlr/v4/tool/ErrorManager.java Outdated Show resolved Hide resolved

parrt reviewed Jun 26, 2022

View reviewed changes

runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTests.java Outdated Show resolved Hide resolved

parrt reviewed Jun 26, 2022

View reviewed changes

runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers/RuntimeTestParser.java Show resolved Hide resolved

parrt reviewed Jun 26, 2022

View reviewed changes

runtime-testsuite/test/org/antlr/v4/test/runtime/FileUtils.java Show resolved Hide resolved

parrt reviewed Jun 26, 2022

View reviewed changes

doc/antlr-project-testing.md Show resolved Hide resolved

KvanTTT added 2 commits June 26, 2022 22:51

Fix potential problems with concurrent access to hash maps

d40067d

Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

Restore info about mvn -Dtest=java.** test in antlr-project-testing.md

de255cc

Fix misprint Signed-off-by: Ivan Kochurkin <kvanttt@gmail.com>

parrt added comp:runtime type:improvement comp:testing labels Jun 26, 2022

parrt added this to the 4.10.2 milestone Jun 26, 2022

parrt merged commit ff2acb1 into antlr:dev Jun 26, 2022

KvanTTT deleted the parallel-lock-free-testing branch June 26, 2022 21:36

KvanTTT mentioned this pull request Jun 27, 2022

Make runtime-testsuite tests run in parallel #3423

Closed

lppedd added a commit to lppedd/antlr-kotlin that referenced this pull request Dec 5, 2023

refactor: fix potential deadlocks in Java runtime (antlr4#3752)

90ef7a5

See antlr/antlr4#3752 Commit 875911c2b2e6e50317e75ff2b5e80f4014a75954

admin-token-bot mentioned this pull request Jul 7, 2024

[Snyk] Upgrade antlr4 from 4.7.2 to 4.13.1 mongodb-js/compass#6001

Closed

ThatGhostToast mentioned this pull request Nov 7, 2024

[Snyk] Upgrade antlr4 from 4.8.0 to 4.13.2 NextechSystems/fhirpath.js#3

Closed

dependabot bot mentioned this pull request Jan 5, 2026

Bump Antlr4.Runtime.Standard from 4.9.3 to 4.13.1 serdedotnet/serde.xml#3

Closed


		All test are run in parallel both via maven and via IDE.

		In IntelliJ, it's very easy to go to source by right-click on any test and pressing `Jump to source` (F4).

Conversation

KvanTTT commented Jun 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

parrt commented Jun 20, 2022

Uh oh!

KvanTTT commented Jun 20, 2022

Uh oh!

parrt commented Jun 20, 2022

Uh oh!

KvanTTT commented Jun 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

parrt commented Jun 24, 2022

Uh oh!

KvanTTT commented Jun 25, 2022

Uh oh!

KvanTTT commented Jun 25, 2022

Uh oh!

KvanTTT commented Jun 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

parrt commented Jun 26, 2022

Uh oh!

Uh oh!

parrt Jun 26, 2022

Choose a reason for hiding this comment

Uh oh!

KvanTTT Jun 26, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KvanTTT commented Jun 26, 2022

Uh oh!

parrt commented Jun 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

KvanTTT commented Jun 19, 2022 •

edited

Loading

KvanTTT commented Jun 20, 2022 •

edited

Loading

KvanTTT commented Jun 25, 2022 •

edited

Loading