SpringBoot Strip empty layers when publishing OCI images

spring-boot-maven-plugin version 3.5.5 produces empty layers in image manifest: looks like:

{"Config":"blobs/sha256/0c06242f93377ecb92405b0bd2e741610c1e6c1c6b5d043df9fe2a5d63967114","RepoTags":["my-service:1.0.0-SNAPSHOT"],"Layers":["blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","blobs/ 
...

The problem is that some docker registries reject this manifest, for example I see the following error in AWS ECR:

error from registry: Layers with digests '[sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855]' required for pushing image into repository with name 'my-service' in the registry with id 'xxxxx' do not exist

I assume that empty layers haven't be produced before. Can it be fixed?

Comment From: wilkinsona

I don't think this is a Spring Boot problem as it's not involved in determining the contents of each layer. It's the buildpacks and builder that do that. Does the problem occur when using the pack CLI?

Comment From: ashakirin

@wilkinsona: no, when I use pack cli, empty layer is not produced and I can import the image into ECR:

pack build ${ECR_URI}:latest \
  --builder paketobuildpacks/builder-noble-java-tiny \
  --tag ${ECR_URI} \
  --env BP_JVM_VERSION=21 \
   --publish

But when I run mvn spring-boot:build-image with default settings, empty layer is in manifest

Comment From: wilkinsona

That's interesting (and surprising). Thanks for trying it. Can you please provide a minimal Maven project that reproduces the problem?

Comment From: wilkinsona

I think I've reproduced this while investigating another problem. It looks like it may be platform specific as I see it on my MacBook Pro (it's an M1 pro) but CI on Linux is unaffected. In my testing, the problem only occurs with paketobuildpacks/builder-noble-java-tiny:0.0.56 and later so it appears to be a regression in the builder.

@ashakirin does the above match your situation at all? Also, please try paketobuildpacks/builder-noble-java-tiny:0.0.55.

Comment From: ashakirin

we experience this problem on Amazon Linux as well, (6.1.147-172.266.amzn2023.x86_64 x86_64 GNU/Linux) when creating the image using mvn spring-boot:build-image I attached a small SpringBoot project (including image manifest). If the image is produced with SpringBoot plugin and saved, manifest contains digest of empty layer: blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

demo-image-issue.zip

Comment From: wilkinsona

Thanks. Did you try paketobuildpacks/builder-noble-java-tiny:0.0.55?

Comment From: ashakirin

Yes, if I build the image with command ./pack build ashakirin/unicorn-spring-ai-agent:latest \ --builder paketobuildpacks/builder-noble-java-tiny:0.0.56 \ --env BP_JVM_VERSION=21 \ --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar --publish the result image manifest DOESN'T contain empty layers with digest blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 .

Pack CLI image manifest is attached, you can compare it with first one from maven plugin build.

manifest.json

Comment From: philwebb

I think I can recreate this issue with pack after running mvn package on the sample:

$ docker system prune --all
$ pack build demo-image-issue --builder paketobuildpacks/builder-noble-java-tiny:latest --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar
$ docker image save demo-image-issue | tar -xf - manifest.json && cat manifest.json | jq

Gives:

[
  {
    "Config": "blobs/sha256/0a05309c5285563eb3091f85153f5347c021a4b0d91ae9ec39583d0aa3be6014",
    "RepoTags": [
      "demo-image-issue:latest"
    ],
    "Layers": [
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "blobs/sha256/3e74b876fa888a7265efcbe547e07018ccda0421797d9ae6cbf185d6d1d5508c",
      "blobs/sha256/e13f32b89503082d9d076df31b0ab4fe367f17308145e2e9777332f52351305f",
      "blobs/sha256/5f4c1d5773eb3a00ebec6070263004ec60faf7311ea4db1ac5ff045aecd165f8",
      "blobs/sha256/a4124ebfe5c4a63a1f5020c530f176f0fb62c2edc2e0c87220cb0bba6d159b74",
      "blobs/sha256/b8578b8a2ac51f7abdfdd4eea28bc34a73920771ba01df61f9f6d30f2bbc0dfe",
      "blobs/sha256/d6ecce2bcb621e3b99b3fadd213faf56a32a4d221855c6726d317a931b97401e",
      "blobs/sha256/2cdf5b962e8ece0e546450ea779424e3420db80186dfafc0a0f44d172cee1c2f",
      "blobs/sha256/bc7f82c871ffa12af89cd62c3047e5289ce24fbbe07f7adeeb840ccd43140f20",
      "blobs/sha256/8e326708c473538285f07c789947ca89fa19fbbd7ae08333ec883c0fcba4d7cc",
      "blobs/sha256/cc3c0e569c3cd1ea2ca983256c6dc1c4ef10fe3c381b1fa0c450925c257dde94",
      "blobs/sha256/24d5d2eaea1e2fc53ad9e081541acb105a2ae62a6dcffb875dec06b573149c0a",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/891c5ef4572e1828d36c43534cacffd20bda39314b768e995bc327f475e2be96",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/34061cc4ad826866fb162f006f0a8e7b98cf4c6d4ae8644335d5fce00c345027",
      "blobs/sha256/f7f72ce47a0b7ddbe653dbe7ae8304e39f8ac3f29a8f16cd0623223373b4936e",
      "blobs/sha256/b39b18afe74ec7ee652aed10c5c3e637af7e7daad3da9380043a93fe7d212d44"
    ]
  }
]

Where as:

$ docker system prune --all
$ pack build demo-image-issue --builder paketobuildpacks/builder-noble-java-tiny:0.0.55 --env BP_JVM_VERSION=21 --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar
$ docker image save demo-image-issue | tar -xf - manifest.json && cat manifest.json | jq

Gives:

[
  {
    "Config": "blobs/sha256/5d6e19b261e7fec3ae6ac29665eeaea85acf8063f2ae7a26b63ddc05fc541583",
    "RepoTags": [
      "demo-image-issue:latest"
    ],
    "Layers": [
      "blobs/sha256/8637476602cf2504187f2da7bc4b67fe11ea77b97d8506d33b791aaeac110e76",
      "blobs/sha256/72c283668f14198365eace6c60701babd1fa267eac2b647fbad0352d0c97af86",
      "blobs/sha256/aac046430871674bd6f2219bb468681c5b3338f1e06ab4c14197bb5885968ad3",
      "blobs/sha256/5b6f8eaa5fccd62d95fffce6fecfd06b35f5fd04417d0412e61339005d028490",
      "blobs/sha256/9f6e2a03234306101ca35e7e361bd3f08fc4d55beeeacd8c351204106e29aae3",
      "blobs/sha256/bea0a3dc2651cac7c9c567a5cb4e7536107b357cb9113e8806f690f050500012",
      "blobs/sha256/03160ce98d2ab327ffbb1a513d2d758f57214b22ffafc035151801da7e84ca5c",
      "blobs/sha256/417e5bfc3c82b9373cf6804206e071d2fc74560df867d0f39cb21ac3d15231b6",
      "blobs/sha256/e8dc1c799ea8bf3d9a939276b9b4a6ae15843738f1c1ef94239dc51b2ea62f3c",
      "blobs/sha256/f0e9078fd509b2493bcef1215b4de825d53f118ef0648b850edec03dc6e3cfd4",
      "blobs/sha256/366ce7d1a7f90f2e4ad08752f87510eee3ffca18736fa63c03823c8c4ebf2925",
      "blobs/sha256/a7dca6c9996468c5b91aec53520bc781895168f949c93815ac501062411c153f",
      "blobs/sha256/f0d0405d31af85d016a44d855faf4fad2bcf23e3b31ebb3c313f4714c7ffeea7",
      "blobs/sha256/92b2ae71f144e493fd8c6172dfd184b430c316ddbda3a565cdfaa47fc667fbc0",
      "blobs/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
      "blobs/sha256/e22c76e803cb583614fe12bc6090b68f2cfdabc0700dca0a97627bbf4226c464",
      "blobs/sha256/89732bc7504122601f40269fc9ddfb70982e633ea9caf641ae45736f2846b004",
      "blobs/sha256/787c1f3510bd8ba56dd3944d217ed6f8a53c6eda5d55a11fbeee1c189dec9a3f",
      "blobs/sha256/dc62fc7eb1a822b0d70fc11a4cbef7a7c804c699374c6c312695cbffd6a995b9",
      "blobs/sha256/1dc94a70dbaa2171fb086500a5d27797f779219b126b0a1eebb9180c2792e80e"
    ]
  }
]

@ashakirin How are you getting the manifest.json you attached? Also, how are you attempting to publish the image? Are you using Maven configuration do publish directly from the build? Perhaps pack is better at stripping out zero length layers when it actually uploads the image.

Comment From: ashakirin

@philwebb: you are right, looks like it make difference either I publish image to the DockerHub or don't: 1. in the first test I publish image to the docker hub with command ./pack build ashakirin/unicorn-spring-ai-agent:latest \ --builder paketobuildpacks/builder-noble-java-tiny:0.0.56 \ --env BP_JVM_VERSION=21 \ --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar --publish, then pull the image locally and check it's manifest: it doesn't contain empty layers digest.

1. in the second test I saved image locally with command ./pack build demo-image-issue:latest \ --builder paketobuildpacks/builder-noble-java-tiny:0.0.56 \ --env BP_JVM_VERSION=21 \ --path ./target/demo-image-issue-0.0.1-SNAPSHOT.jar and analysed local image manifest (attached) and it contains empty layers.

Resume: it looks like buildpack issue, not the Spring Boot plugin. The fact, that behavior seems to be changed in last releases, because I am sure, it works before.

manifest.json

Comment From: philwebb

Thanks, I think we can perhaps update our publishing code to strip empty layers.

@ashakirin So you use <publish>true</publish> in your pom.xml or do you publish some other way?

Comment From: philwebb

I've opened https://github.com/paketo-buildpacks/ubuntu-noble-builder/issues/52

Comment From: wilkinsona

I'm not sure that we should strip out the empty layers. It feels like a bug in the builder that we'd be masking. That could add to some of the confusion that we saw here where it worked fine in some usages of pack but not with our build plugins or a different usage of pack. That's heading down a road where you have to use our tooling for the publish step and nothing else which I think we might come to regret. I'd prefer that we at least wait for the response to https://github.com/paketo-buildpacks/ubuntu-noble-builder/issues/52 before doing anything here.

Comment From: ashakirin

yes, I prefer to fix it in clean way in buildpacks, the problem is not urgent @wilkinsona: thanks a lot for openning the buildpacks issue and quick reaction!

Comment From: paulux84

i've tried builder-noble-java-tiny:0.0.55 and works correctly

Comment From: krnbr

Adding the comment here only for better tracing across globe.

Generally - One should be able to build image on local, should be able to push too.

But because of the issue reported above, one may get the following error while pulling it on a remote machine.

The error would be = layers from manifest don't match image configuration

for the time being only option was to move back to builder - paketobuildpacks/builder-noble-java-tiny:0.0.55 as pointed out by @wilkinsona

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <image>
            <name>org/image-name:${project.version}</name>
            <createdDate>now</createdDate>
            <builder>**paketobuildpacks/builder-noble-java-tiny:0.0.55**</builder>
            <publish>true</publish>
            <cleanCache>true</cleanCache>
        </image>
        <excludes>
            <exclude>
                <groupId>org.projectlombok</groupId>
                <artifactId>lombok</artifactId>
            </exclude>
        </excludes>
    </configuration>
</plugin>