Static memory snapshot collecting and restoring #418

dmitrii-artuhov · 2024-10-17T23:28:01Z

This PR partly resolves issue #389

Here we introduced a lazy-tracking feature which aims to only track and restore variables reachable from static memory iff the variables were accessed perviously via read or write operations. Yet, there are still problems (see issue) that are left to be tackled in separate PRs.

Proposed solutions in this PR:

lazy tracking for most of the variables
snapshot of collections are stored enegetically (meaning that the whole object graph is snapshotted). There are several APIs that are not handled yet, which does not allow a correct lazy snapshot building (e.g. System.arraycopy(...))
leaking this problem: I added a fix, which allows the test mentioned in the issue to pass, see. The TODO (calculating common superclass without loading additional classes to the VM) was resolved by leveraging SafeClassWriter::getCommonSuperClass(...). Thanks to @ndkoval for help!

eupp · 2024-11-26T16:17:55Z

Let's rebase on develop (part of the diff should also disappear, because it is already merged into develop).

…teger', etc)

...m/main/org/jetbrains/kotlinx/lincheck/strategy/managed/modelchecking/ModelCheckingCTest.java

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/managed/ManagedStrategy.kt

bootstrap/src/sun/nio/ch/lincheck/EventTracker.java

...ain/org/jetbrains/kotlinx/lincheck/transformation/transformers/SnapshotTrackerTransformer.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/managed/SnapshotTracker.kt

src/jvm/test/resources/expected_logs/var_handle/varhandle_boolean_representation.txt

.../org/jetbrains/kotlinx/lincheck_test/strategy/modelchecking/snapshot/AtomicFUSnapshotTest.kt

eupp

@dmitrii-artuhov Please fix a few remaining minor issues, otherwise looks good to me.

src/jvm/main/org/jetbrains/kotlinx/lincheck/Utils.kt

bootstrap/src/sun/nio/ch/lincheck/EventTracker.java

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/managed/ManagedCTestConfiguration.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/managed/ManagedOptions.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/managed/SnapshotTracker.kt

.../org/jetbrains/kotlinx/lincheck/transformation/transformers/SharedMemoryAccessTransformer.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/util/ObjectGraph.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/util/UnsafeHolder.kt

eupp · 2024-12-02T15:52:49Z

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/Strategy.kt

    }
-    return null
+
+    restoreStaticMemorySnapshot()


But do you need additional call to restoreStaticMemorySnapshot here, if there is already a call to it in ManagedStrategy::run ?

dmitrii-artuhov · 2024-12-02T16:17:39Z

But do you need additional call to restoreStaticMemorySnapshot here, if there is already a call to it in ManagedStrategy::run?

@eupp I've rewritten it, see the modified Strategy.runIteration(...) function, please

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/Strategy.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/transformation/SafeClassWriter.java

ndkoval · 2024-12-10T20:23:56Z

src/jvm/main/org/jetbrains/kotlinx/lincheck/transformation/LincheckClassVisitor.kt

@@ -91,6 +90,14 @@ internal class LincheckClassVisitor(
        }
        if (methodName == "<init>") {
            mv = ObjectCreationTransformer(fileName, className, methodName, mv.newAdapter())
+            mv = run {


Can we treat <init> similar to normal functions?

I did not try including <init> methods in the whole transformation chain. Right now I only require tracking fields/arrays accesses and other constructor invocations inside <init> method bodies.
Maybe the feature of integrating constructors in full transformation chain should be a separate PR, since there are might be tricky spots where to account for #424

...tlinx/lincheck/transformation/transformers/ConstructorArgumentsSnapshotTrackerTransformer.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/managed/SnapshotTracker.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/managed/ManagedStrategy.kt

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/Strategy.kt

dmitrii-artuhov marked this pull request as draft October 17, 2024 23:28

ndkoval requested a review from eupp November 26, 2024 15:38

ndkoval changed the base branch from trace-debugger to develop November 26, 2024 16:16

dmitrii-artuhov force-pushed the trace-debugger-static-memory branch from c77bef2 to dd0160d Compare November 27, 2024 12:27

dmitrii-artuhov mentioned this pull request Nov 27, 2024

Improve support of static initialization blocks and statically-reachable memory #389

Open

dmitrii-artuhov added 19 commits November 27, 2024 15:33

Add static memory snapshot collecting and restoring

8b73ca2

Add the class which implements the logic for static memory snapshot

ca7d02a

Add tests and fixes to snapshot algorithm

75126a6

Remove comments

3e6ec07

Make snapshot algorithm lazy

760b076

Fix bugs

7edbd56

Add energetic traverse for non-transformed classes (such as 'AtomicIn…

feefb3d

…teger', etc)

Refactor code

128050a

Add energetic snapshot recording for non-transformed classes

c9f7ee2

Enable snapshot restoring for all tests, fix representation tests

b7a488d

Add minor refactoring

b733f25

Add tests for java atomics and atomicfu classes for snapshot restoring

c187f06

Add collections tests

5ef2024

Add tracking for fields of objects the same type as constructor owner

4c08faa

Optimize tracking of constructors' arguments

eb9a9a2

Remove println's and comments

0803b6d

Fix todo with subclass check in SnapshotTrackerTransformer

1b6bbef

Add VarHandles to excludes in heirarchy tracking

86d5bd2

Clean up code (remove comments and useless annotations)

8eac5dc

dmitrii-artuhov force-pushed the trace-debugger-static-memory branch from 336e12d to 8eac5dc Compare November 27, 2024 14:36

dmitrii-artuhov marked this pull request as ready for review November 27, 2024 16:07

ndkoval reviewed Nov 27, 2024

View reviewed changes

...m/main/org/jetbrains/kotlinx/lincheck/strategy/managed/modelchecking/ModelCheckingCTest.java Outdated Show resolved Hide resolved

ndkoval reviewed Nov 27, 2024

View reviewed changes

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/managed/ManagedStrategy.kt Outdated Show resolved Hide resolved

eupp requested changes Nov 29, 2024

View reviewed changes

dmitrii-artuhov added 4 commits December 2, 2024 14:17

Add some PR fixes

709a6d3

Add more fixes to the PR

7f5e6ea

Simplify and refactor code in SnapshotTracker

a15f645

Rewrite values restoring in a non-recursion way using stack

28b153b

dmitrii-artuhov requested a review from eupp December 2, 2024 14:12

eupp reviewed Dec 2, 2024

View reviewed changes

Add fixes to the PR

af39fd7

dmitrii-artuhov requested a review from eupp December 2, 2024 16:19

Add fixes

d8bf525

eupp approved these changes Dec 2, 2024

View reviewed changes

dmitrii-artuhov requested a review from ndkoval December 2, 2024 17:07

ndkoval requested changes Dec 10, 2024

View reviewed changes

dmitrii-artuhov added 3 commits December 11, 2024 15:46

Add PR fixes

3097c43

Bring gradle folder back

fc57dde

Add small renaming

be5329a

dmitrii-artuhov requested a review from ndkoval December 11, 2024 19:26

ndkoval requested changes Dec 11, 2024

View reviewed changes

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/Strategy.kt Outdated Show resolved Hide resolved

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/Strategy.kt Outdated Show resolved Hide resolved

Add minor fixes

6ca73a9

dmitrii-artuhov requested a review from ndkoval December 12, 2024 07:36

ndkoval approved these changes Dec 12, 2024

View reviewed changes

ndkoval merged commit a736006 into develop Dec 12, 2024
16 checks passed

ndkoval deleted the trace-debugger-static-memory branch December 12, 2024 11:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Static memory snapshot collecting and restoring #418

Static memory snapshot collecting and restoring #418

dmitrii-artuhov commented Oct 17, 2024 •

edited

Loading

eupp commented Nov 26, 2024 •

edited

Loading

eupp left a comment

eupp Dec 2, 2024

dmitrii-artuhov commented Dec 2, 2024 •

edited

Loading

ndkoval Dec 10, 2024

dmitrii-artuhov Dec 11, 2024

Static memory snapshot collecting and restoring #418

Static memory snapshot collecting and restoring #418

Conversation

dmitrii-artuhov commented Oct 17, 2024 • edited Loading

eupp commented Nov 26, 2024 • edited Loading

eupp left a comment

Choose a reason for hiding this comment

eupp Dec 2, 2024

Choose a reason for hiding this comment

dmitrii-artuhov commented Dec 2, 2024 • edited Loading

ndkoval Dec 10, 2024

Choose a reason for hiding this comment

dmitrii-artuhov Dec 11, 2024

Choose a reason for hiding this comment

dmitrii-artuhov commented Oct 17, 2024 •

edited

Loading

eupp commented Nov 26, 2024 •

edited

Loading

dmitrii-artuhov commented Dec 2, 2024 •

edited

Loading