Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it easier to test against multiple versions of the JDK #1072

Open
RyanGlScott opened this issue Feb 11, 2021 · 3 comments
Open

Make it easier to test against multiple versions of the JDK #1072

RyanGlScott opened this issue Feb 11, 2021 · 3 comments
Labels
subsystem: crucible-jvm Issues related to Java verification with crucible-jvm tooling: test infrastructure Issues involving test infrastructure or test execution, or making SAW more testable type: feature request Issues requesting a new feature or capability
Milestone

Comments

@RyanGlScott
Copy link
Contributor

In my work on #1046, I briefly considered extending saw-script's CI so that in addition to testing JDK 8, it would also test a newer JDK version (e.g., JDK 15) that exercised some new code paths in saw-script. Ultimately, I didn't pursue this direction too deeply for two reasons:

  1. Support for modern JDKs is still experimental in light of crucible-jvm: Investigate bugs when verifying JDK 9+ code that uses String crucible#641.
  2. The naïve way of testing multiple JDKs—namely, doubling the number of jobs such that half of them test JDK 8 and the other half test JDK 15—would result in a lot of wasted work, as we only really need to re-run the parts of CI that exercise Java–specific code paths. For example, we really only ought to run LLVM-related tests once overall, not once per JDK version.

GaloisInc/crucible#641 tracks (1), so this issue exists to track (2). The way I see it, we should do something like this:

  • Investigate if setup-java makes it possible to simultaneously have multiple versions of the JDK installed side by side, allowing one to switch between one version or the other. My hope is that setup-java allows for something akin to update-java-alternatives, but I'm not sure if that's the case off the top of my head.
  • Clearly indicate in the test suite which tests use Java and which do not so that we can parameterize the Java-specific tests by the particular JDK version being tested against. That way, we need only re-run the Java specific tests if we support multiple JDKs, and we can avoid re-running other tests such as LLVM-specific tests.
@RyanGlScott RyanGlScott added subsystem: crucible-jvm Issues related to Java verification with crucible-jvm tooling: test infrastructure Issues involving test infrastructure or test execution, or making SAW more testable labels Feb 11, 2021
@sauclovian-g sauclovian-g added the type: feature request Issues requesting a new feature or capability label Oct 30, 2024
@sauclovian-g sauclovian-g added this to the Someday milestone Oct 30, 2024
@sauclovian-g
Copy link
Contributor

It looks to me like at least some of the Java tests in intTests have accompanying checked-in class files, and the rest presumably could. Is there some runtime dependency of Java verification on a JVM? Because otherwise it seems like the path of least resistance is just compiling and checking in multiple versions.

@RyanGlScott
Copy link
Contributor Author

Is there some runtime dependency of Java verification on a JVM?

Sadly, yes. You need to access the Java standard library separately, as the relevant class definitions (e.g., for String) aren't included in the checked-in class files. I wrote up a lengthier explanation of how this works on the crucible-jvm side here.

@sauclovian-g
Copy link
Contributor

I suppose that makes sense. Bleh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
subsystem: crucible-jvm Issues related to Java verification with crucible-jvm tooling: test infrastructure Issues involving test infrastructure or test execution, or making SAW more testable type: feature request Issues requesting a new feature or capability
Projects
None yet
Development

No branches or pull requests

2 participants