The spring boot maven plugin rewrites the jars that a build creates when executing the spring-boot:process-test-aot goal.

This goal is dependent on the maven-wide 'skipTests' and 'maven.test.skip' properties. If either of these is set, the execution is skipped.

For modern java projects, it is common to provide reproducible builds (See https://reproducible-builds.org/docs/jvm/ for details). In a reproducible build cycle, the CI does

  • a full build of a project executing all tests (neither 'skipTests' nor 'maven.test.skip' is set). The resulting jars are installed in the local repository

  • a second build using (for maven) the artifact:compare target. For that second build, usually tests and other checks are skipped as the goal of that build is simply to validate that the resulting jars are bitwise identical.

Currently, it is impossible to use the spring-boot:process-test-aot in this reproducible build cycle. Adding the plugin to the build results in the plugin being executed in the first build, but skipped in the second as it looks at the skipTests and maven.test.skip which are set to actually skip the tests while the spring-boot target should be executed independent of whether the tests are run or not.

Not being able to control the execution of this goal independent of the actual test execution (whether through maven-surefire, maven-native or any other maven test plugin) is a bug. This goal is not a test specific goal but a test-build specific goal.

The current situation makes it impossible for to provide reproducible builds if one wants to support native builds and also support spring. Dropping native build support is not an option.

One possible solution is making the execution of the spring-boot:process-test-aot not directly dependent on the properties mentioned above but have a configuration option to skip or run the goal and have its default set from the proprties but allow users to override this in the maven pom file. This would allow for reproducible builds while maintaining full compatibility to the current functionality.

Comment From: wilkinsona

Currently, it is impossible to use the spring-boot:process-test-aot in this reproducible build cycle. Adding the plugin to the build results in the plugin being executed in the first build, but skipped in the second

Why does that matter in the context of artifact:compare? I understand the benefits of a build's "production" artifacts being reproducible but does that comparison consider test classes (and possibly a test-specific native image too)?

Comment From: hgschmie

because the artifact gets published (like all other maven artifacts). Users tend to take a dim view if we tell them "you can verify these 34 artifacts but not those three". :-)

In this specific case, we are shipping an (OSS) library to support spring as part of our project (https://jdbi.org/#_spring_framework). We have > 200 artifacts and all of them are reproducible for regular and native builds. Except the spring test jars.

Comment From: wilkinsona

Publishing tests is, in my experience, rather unusual. That would have been useful information to provide up front.

To help us to understand exactly what you're doing, please provide a minimal sample that exhibits the non-reproducible behavior that you've described.

Comment From: hgschmie

Thanks. Not publishing the test jars is a good point, something to think about. I am not sure whether artifact:compare can skip jars that will be installed but not deployed (it may need some additional logic). We do need to install the jars locally because we have integration tests that depend on some of the test jars.

Skipping them for some sub-projects may be an option. The wider issue (that the spring-boot:process-test-aot is a goal that is run as part of the test build process and not the test execution process and therefore should be available if tests are built but not executed) still remains, even though we may find ways to work around this.

Comment From: wilkinsona

We can't make any progress here without understanding exactly how you've set up Maven. To that end, please provide a minimal sample that exhibits the non-reproducible behavior that you've described.

Comment From: snicoll

I assume you have some acquaintance with @stevenschlansker? If so, linking the two requests upfront would have been appreciated.

This goal is dependent on the maven-wide 'skipTests' and 'maven.test.skip' properties. If either of these is set, the execution is skipped.

TIL. So I wrote this in the related PR:

it makes no sense for us to bind to a surefire plugin-specify property, especially when our plugin doesn't do anything with test execution. The maven.test.skip property is more general and honored by at least Surefire, Failsafe and the Compiler Plugins. It then makes sense to support it since our process-test-aot is compiling generated test code.

The description in this PR breaks my brain and I now understand with the above description what you're after. I agree that the plugin should not check for skipTests, so I've reopened the PR for further consideration. It probably is incomplete as we'd like to update the doc and the test suite but I can do that as part of polishing.

@wilkinsona I am not sure if that answers your question, but reacting to a surefire-specific property for this goal feels wrong to me.

Comment From: wilkinsona

Thanks, @snicoll.

I am not sure if that answers your question.

Not fully. I'd like to see exactly how Maven has been configured to install the test jars and how the artifact:compare goal has been configured.

Comment From: hgschmie

@wilkinsona The project in question is Jdbi (https://github.com/jdbi/jdbi). @stevenschlansker is putting together a test case that shows the breakage right now. If you want to see how we run our reproducible builds, the simplest way is to check out the repo above and run

make install-fast make compare-reproducible

Note, that the default build (from the master branch) does not use the spring boot aot plugin yet (as we are still ramping up our native effort).

Comment From: wilkinsona

Thanks. Unfortunately, Jdbi's build is quite a bit larger and more complex than we have time for at the moment. Hopefully @stevenschlansker can share something minimal that reproduces the problem.

Comment From: stevenschlansker

I set up a more minimal example: https://github.com/stevenschlansker/spring-boot-aot-reproducible/tree/master

There is a build.sh script which will build the jar both with tests run and without tests run, then diff the tests jar.

When the jar is built without running tests, the test jar does not include AOT processed classes. The plugin does not attempt to run. And, if it did, we probably need to pass in the root configuration classes for aot processing more explicitly since the test runner does not invoke SpringApplication.run like the listener expects.