Repository navigation
Cross JRE testing matrix #335
Description
Use Case
We should run our tests (especially, tests in the instrumentation module) on multiple JVMs after compiling them with JDK 11. This was first brought up on #334 in this comment.
Proposal
One possible method of doing this is leveraging Gradle toolchains. Gradle toolchains support registering Test tasks that require certain JDK versions (see Specify custom toolchains for individual tasks). This has the additional benefit of working on developer's machines as well. Notably, if the toolchain is not detected on the developer's machine, Gradle will automatically download a matching toolchain from AdoptOpenJDK
Questions to address (if any)
There may be other methods of accomplishing this level of cross-jvm testing with Gradle, such as using the GitHub Actions setup-java matrix. But, it's not clear to me how this would be implemented effectively while still compiling with JDK 11 so we can detect niche bytecode compatibility bugs. In addition, depending on the setup, it could be difficult for developers to run these tests locally to debug errors. However, without a specific example test workflow in mind, it is difficult to fairly compare alternate solutions. If folks have code snippets/examples of how this setup could work in practice, please feel free to propose them here
Activity
pavolloffay commentedon Jul 13, 2021
Our tested deliverable is bytecode compiled on JVM 11 into JVM 8 compatible bytecode. What we have to test is that if the deliverable works on other JVMs - all TLS versions and the latest released version.
ryandens commentedon Jul 13, 2021
Yeah, I'm aware of that, the solution I'm proposing accomplishes that using Gradle toolchains, allowing us to define a matrix of JVMs to test. What I'm trying to understand is what your solution would look like in practice
pavolloffay commentedon Jul 13, 2021
I think you got it right, I would use gradle toolchains and matrix configuration on CI.